Oct 30 19:41:26.044070 ip-10-0-142-240 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:41:26.399061 ip-10-0-142-240 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:41:26.399061 ip-10-0-142-240 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:41:26.399061 ip-10-0-142-240 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:41:26.399061 ip-10-0-142-240 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:41:26.402730 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402652    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:41:26.402730 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402731    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:41:26.402809 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402735    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:41:26.402809 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402739    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:41:26.402809 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402742    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:41:26.402809 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402745    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:41:26.402809 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402748    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:41:26.402809 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402751    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:41:26.402809 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402754    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:41:26.402809 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402758    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:41:26.402809 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402762    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:41:26.402809 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402765    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:41:26.402809 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402781    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:41:26.402809 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402785    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:41:26.402809 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402788    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:41:26.402809 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402790    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:41:26.402809 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402793    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:41:26.402809 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402795    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:41:26.402809 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402798    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:41:26.402809 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402800    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:41:26.402809 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402803    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:41:26.403263 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402806    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:41:26.403263 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402814    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:41:26.403263 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402817    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:41:26.403263 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402820    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:41:26.403263 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402823    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:41:26.403263 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402825    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:41:26.403263 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402828    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:41:26.403263 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402831    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:41:26.403263 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402834    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:41:26.403263 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402836    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:41:26.403263 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402839    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:41:26.403263 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402841    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:41:26.403263 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402844    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:41:26.403263 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402847    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:41:26.403263 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402850    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:41:26.403263 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402852    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:41:26.403263 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402855    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:41:26.403263 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402857    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:41:26.403263 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402861    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:41:26.403263 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402863    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:41:26.403709 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402866    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:41:26.403709 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402868    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:41:26.403709 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402871    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:41:26.403709 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402873    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:41:26.403709 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402876    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:41:26.403709 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402878    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:41:26.403709 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402880    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:41:26.403709 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402883    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:41:26.403709 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402885    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:41:26.403709 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402888    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:41:26.403709 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402890    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:41:26.403709 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402894    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:41:26.403709 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402897    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:41:26.403709 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402900    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:41:26.403709 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402902    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:41:26.403709 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402906    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:41:26.403709 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402911    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:41:26.403709 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402914    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:41:26.403709 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402916    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:41:26.404158 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402919    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:41:26.404158 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402922    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:41:26.404158 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402925    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:41:26.404158 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402927    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:41:26.404158 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402929    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:41:26.404158 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402932    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:41:26.404158 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402934    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:41:26.404158 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402937    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:41:26.404158 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402939    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:41:26.404158 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402942    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:41:26.404158 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402944    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:41:26.404158 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402953    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:41:26.404158 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402957    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:41:26.404158 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402959    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:41:26.404158 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402962    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:41:26.404158 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402965    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:41:26.404158 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402968    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:41:26.404158 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402971    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:41:26.404158 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402973    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:41:26.404590 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402976    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:41:26.404590 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402979    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:41:26.404590 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402982    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:41:26.404590 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402985    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:41:26.404590 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402987    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:41:26.404590 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402990    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:41:26.404590 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402992    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:41:26.404590 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402995    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:41:26.404590 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.402998    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:41:26.404590 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.403001    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:41:26.404590 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.403003    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:41:26.404590 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.403006    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:41:26.404590 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.403009    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:41:26.404590 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.403011    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:41:26.404590 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.403014    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:41:26.404590 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.403016    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:41:26.404590 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.403019    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:41:26.404590 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.403021    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:41:26.404590 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.403024    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:41:26.404590 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.403026    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:41:26.405051 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.403028    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:41:26.405051 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.403031    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:41:26.405051 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.403033    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:41:26.405051 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.403036    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:41:26.405051 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.403038    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:41:26.405051 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.403041    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:41:26.405051 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.403044    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:41:26.405051 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404359    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:41:26.405051 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404368    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:41:26.405051 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404372    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:41:26.405051 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404375    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:41:26.405051 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404379    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:41:26.405051 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404382    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:41:26.405051 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404385    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:41:26.405051 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404388    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:41:26.405051 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404391    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:41:26.405051 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404394    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:41:26.405051 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404398    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:41:26.405051 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404402    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:41:26.405521 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404405    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:41:26.405521 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404408    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:41:26.405521 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404411    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:41:26.405521 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404413    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:41:26.405521 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404416    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:41:26.405521 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404419    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:41:26.405521 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404422    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:41:26.405521 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404425    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:41:26.405521 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404427    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:41:26.405521 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404430    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:41:26.405521 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404433    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:41:26.405521 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404436    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:41:26.405521 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404438    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:41:26.405521 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404441    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:41:26.405521 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404444    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:41:26.405521 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404452    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:41:26.405521 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404456    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:41:26.405521 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404460    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:41:26.405521 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404463    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:41:26.405982 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404466    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:41:26.405982 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404469    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:41:26.405982 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404472    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:41:26.405982 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404474    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:41:26.405982 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404478    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:41:26.405982 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404481    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:41:26.405982 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404483    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:41:26.405982 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404486    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:41:26.405982 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404489    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:41:26.405982 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404492    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:41:26.405982 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404494    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:41:26.405982 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404497    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:41:26.405982 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404499    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:41:26.405982 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404502    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:41:26.405982 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404505    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:41:26.405982 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404507    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:41:26.405982 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404510    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:41:26.405982 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404513    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:41:26.405982 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404515    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:41:26.406422 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404518    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:41:26.406422 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404521    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:41:26.406422 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404523    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:41:26.406422 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404526    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:41:26.406422 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404528    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:41:26.406422 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404531    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:41:26.406422 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404533    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:41:26.406422 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404536    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:41:26.406422 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404538    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:41:26.406422 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404541    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:41:26.406422 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404543    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:41:26.406422 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404546    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:41:26.406422 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404549    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:41:26.406422 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404551    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:41:26.406422 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404554    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:41:26.406422 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404557    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:41:26.406422 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404559    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:41:26.406422 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404562    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:41:26.406422 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404565    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:41:26.406869 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404568    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:41:26.406869 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404571    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:41:26.406869 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404575    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:41:26.406869 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404577    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:41:26.406869 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404580    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:41:26.406869 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404582    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:41:26.406869 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404585    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:41:26.406869 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404587    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:41:26.406869 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404590    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:41:26.406869 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404592    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:41:26.406869 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404595    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:41:26.406869 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404598    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:41:26.406869 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404601    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:41:26.406869 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404603    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:41:26.406869 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404606    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:41:26.406869 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404608    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:41:26.406869 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404611    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:41:26.406869 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404614    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:41:26.406869 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404616    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:41:26.406869 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404619    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:41:26.407323 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404621    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:41:26.407323 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404624    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:41:26.407323 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404626    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:41:26.407323 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404629    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:41:26.407323 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404631    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:41:26.407323 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404633    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:41:26.407323 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404636    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:41:26.407323 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404638    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:41:26.407323 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404641    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:41:26.407323 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404645    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:41:26.407323 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404647    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:41:26.407323 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404650    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:41:26.407323 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404652    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:41:26.407323 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404655    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:41:26.407323 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404658    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:41:26.407323 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404660    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:41:26.407323 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.404663    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:41:26.407323 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405150    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:41:26.407323 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405156    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:41:26.407323 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405160    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:41:26.407851 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405162    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:41:26.407851 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405165    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:41:26.407851 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405168    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:41:26.407851 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405171    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:41:26.407851 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405174    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:41:26.407851 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405177    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:41:26.407851 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405179    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:41:26.407851 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405182    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:41:26.407851 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405184    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:41:26.407851 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405187    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:41:26.407851 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405190    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:41:26.407851 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405192    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:41:26.407851 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405195    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:41:26.407851 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405197    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:41:26.407851 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405200    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:41:26.407851 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405202    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:41:26.407851 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405205    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:41:26.407851 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405207    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:41:26.407851 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405210    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:41:26.407851 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405212    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:41:26.408308 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405215    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:41:26.408308 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405218    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:41:26.408308 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405220    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:41:26.408308 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405223    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:41:26.408308 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405225    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:41:26.408308 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405228    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:41:26.408308 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405231    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:41:26.408308 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405236    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:41:26.408308 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405239    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:41:26.408308 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405242    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:41:26.408308 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405245    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:41:26.408308 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405247    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:41:26.408308 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405250    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:41:26.408308 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405253    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:41:26.408308 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405255    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:41:26.408308 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405258    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:41:26.408308 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405261    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:41:26.408308 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405264    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:41:26.408308 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405266    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:41:26.408734 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405269    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:41:26.408734 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405272    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:41:26.408734 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405274    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:41:26.408734 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405277    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:41:26.408734 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405279    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:41:26.408734 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405281    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:41:26.408734 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405284    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:41:26.408734 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405286    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:41:26.408734 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405289    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:41:26.408734 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405291    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:41:26.408734 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405294    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:41:26.408734 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405297    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:41:26.408734 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405299    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:41:26.408734 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405302    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:41:26.408734 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405304    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:41:26.408734 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405307    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:41:26.408734 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405310    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:41:26.408734 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405312    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:41:26.408734 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405315    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:41:26.408734 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405317    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:41:26.409208 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405320    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:41:26.409208 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405323    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:41:26.409208 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405325    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:41:26.409208 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405328    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:41:26.409208 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405330    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:41:26.409208 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405333    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:41:26.409208 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405335    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:41:26.409208 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405338    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:41:26.409208 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405341    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:41:26.409208 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405343    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:41:26.409208 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405346    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:41:26.409208 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405348    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:41:26.409208 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405351    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:41:26.409208 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405353    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:41:26.409208 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405355    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:41:26.409208 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405358    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:41:26.409208 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405360    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:41:26.409208 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405363    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:41:26.409208 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405366    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:41:26.409653 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405368    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:41:26.409653 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405370    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:41:26.409653 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405373    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:41:26.409653 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405376    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:41:26.409653 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405378    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:41:26.409653 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405381    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:41:26.409653 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405383    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:41:26.409653 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405386    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:41:26.409653 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405388    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:41:26.409653 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405391    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:41:26.409653 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405394    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:41:26.409653 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405399    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:41:26.409653 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405401    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:41:26.409653 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405404    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:41:26.409653 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405406    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:41:26.409653 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405409    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:41:26.409653 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405412    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:41:26.409653 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405414    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:41:26.409653 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405418    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:41:26.409653 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405422    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:41:26.410129 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405425    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:41:26.410129 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405429    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:41:26.410129 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405431    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:41:26.410129 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405434    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:41:26.410129 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.405436    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:41:26.410129 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405517    2573 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:41:26.410129 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405523    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:41:26.410129 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405529    2573 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:41:26.410129 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405533    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:41:26.410129 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405538    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:41:26.410129 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405541    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:41:26.410129 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405545    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:41:26.410129 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405549    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:41:26.410129 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405552    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:41:26.410129 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405554    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:41:26.410129 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405558    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:41:26.410129 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405561    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:41:26.410129 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405563    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:41:26.410129 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405566    2573 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:41:26.410129 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405569    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:41:26.410129 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405572    2573 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:41:26.410129 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405574    2573 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:41:26.410129 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405578    2573 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:41:26.410129 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405604    2573 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:41:26.410695 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405615    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:41:26.410695 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405619    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:41:26.410695 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405622    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:41:26.410695 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405626    2573 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:41:26.410695 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405630    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:41:26.410695 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405633    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:41:26.410695 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405636    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:41:26.410695 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405639    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:41:26.410695 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405642    2573 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:41:26.410695 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405645    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:41:26.410695 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405648    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:41:26.410695 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405651    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:41:26.410695 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405653    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:41:26.410695 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405657    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:41:26.410695 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405660    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:41:26.410695 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405663    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:41:26.410695 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405666    2573 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:41:26.410695 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405668    2573 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:41:26.410695 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405671    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:41:26.410695 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405675    2573 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:41:26.410695 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405678    2573 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:41:26.410695 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405681    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:41:26.410695 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405683    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:41:26.410695 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405686    2573 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:41:26.411273 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405691    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:41:26.411273 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405693    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:41:26.411273 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405697    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:41:26.411273 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405699    2573 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:41:26.411273 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405702    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:41:26.411273 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405705    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:41:26.411273 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405707    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:41:26.411273 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405714    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:41:26.411273 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405717    2573 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:41:26.411273 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405720    2573 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:41:26.411273 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405723    2573 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:41:26.411273 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405727    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:41:26.411273 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405730    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:41:26.411273 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405733    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:41:26.411273 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405736    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:41:26.411273 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405739    2573 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:41:26.411273 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405742    2573 flags.go:64] FLAG: --help="false"
Oct 30 19:41:26.411273 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405746    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:41:26.411273 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405749    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:41:26.411273 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405751    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:41:26.411273 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405754    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:41:26.411273 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405757    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:41:26.411273 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405761    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:41:26.411987 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405763    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:41:26.411987 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405766    2573 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:41:26.411987 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405784    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:41:26.411987 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405787    2573 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:41:26.411987 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405790    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:41:26.411987 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405793    2573 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:41:26.411987 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405795    2573 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:41:26.411987 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405798    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:41:26.411987 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405801    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:41:26.411987 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405803    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:41:26.411987 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405806    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:41:26.411987 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405809    2573 flags.go:64] FLAG: --lock-file=""
Oct 30 19:41:26.411987 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405812    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:41:26.411987 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405814    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:41:26.411987 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405817    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:41:26.411987 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405822    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:41:26.411987 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405826    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:41:26.411987 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405832    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:41:26.411987 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405837    2573 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:41:26.411987 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405840    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:41:26.411987 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405843    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:41:26.411987 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405846    2573 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:41:26.411987 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405849    2573 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:41:26.411987 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405853    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:41:26.411987 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405856    2573 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405860    2573 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405863    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405865    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405868    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405871    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405874    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405876    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405879    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405886    2573 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405889    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405892    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405895    2573 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405897    2573 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405900    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405903    2573 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405905    2573 flags.go:64] FLAG: --port="10250"
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405908    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405911    2573 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0901a26e1f416de69"
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405914    2573 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405917    2573 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405920    2573 flags.go:64] FLAG: --register-node="true"
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405923    2573 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405926    2573 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405930    2573 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405933    2573 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:41:26.413029 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405937    2573 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:41:26.414121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405941    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:41:26.414121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405944    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:41:26.414121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405947    2573 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:41:26.414121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405950    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:41:26.414121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405952    2573 flags.go:64] FLAG: --runonce="false"
Oct 30 19:41:26.414121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405955    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:41:26.414121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405958    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:41:26.414121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405961    2573 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:41:26.414121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405964    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:41:26.414121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405967    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:41:26.414121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405970    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:41:26.414121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405973    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:41:26.414121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405976    2573 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:41:26.414121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405978    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:41:26.414121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405981    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:41:26.414121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405984    2573 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:41:26.414121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405986    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:41:26.414121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405989    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:41:26.414121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405992    2573 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:41:26.414121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.405995    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:41:26.414121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.406000    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:41:26.414121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.406003    2573 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:41:26.414121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.406006    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:41:26.414121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.406010    2573 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:41:26.414121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.406013    2573 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:41:26.415174 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.406016    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:41:26.415174 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.406019    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:41:26.415174 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.406021    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:41:26.415174 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.406024    2573 flags.go:64] FLAG: --v="2"
Oct 30 19:41:26.415174 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.406029    2573 flags.go:64] FLAG: --version="false"
Oct 30 19:41:26.415174 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.406033    2573 flags.go:64] FLAG: --vmodule=""
Oct 30 19:41:26.415174 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.406038    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:41:26.415174 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.406041    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:41:26.415174 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406170    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:41:26.415174 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406175    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:41:26.415174 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406177    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:41:26.415174 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406180    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:41:26.415174 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406182    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:41:26.415174 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406185    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:41:26.415174 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406188    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:41:26.415174 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406190    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:41:26.415174 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406193    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:41:26.415174 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406195    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:41:26.415174 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406198    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:41:26.415174 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406200    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:41:26.415174 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406203    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:41:26.415174 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406205    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:41:26.416162 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406208    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:41:26.416162 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406211    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:41:26.416162 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406213    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:41:26.416162 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406216    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:41:26.416162 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406218    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:41:26.416162 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406221    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:41:26.416162 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406223    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:41:26.416162 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406226    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:41:26.416162 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406228    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:41:26.416162 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406231    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:41:26.416162 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406233    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:41:26.416162 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406236    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:41:26.416162 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406238    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:41:26.416162 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406241    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:41:26.416162 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406245    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:41:26.416162 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406249    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:41:26.416162 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406253    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:41:26.416162 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406256    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:41:26.416162 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406259    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:41:26.416901 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406261    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:41:26.416901 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406264    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:41:26.416901 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406266    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:41:26.416901 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406269    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:41:26.416901 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406271    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:41:26.416901 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406274    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:41:26.416901 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406277    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:41:26.416901 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406279    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:41:26.416901 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406282    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:41:26.416901 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406284    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:41:26.416901 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406287    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:41:26.416901 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406289    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:41:26.416901 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406292    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:41:26.416901 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406294    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:41:26.416901 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406297    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:41:26.416901 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406299    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:41:26.416901 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406302    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:41:26.416901 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406304    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:41:26.416901 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406307    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:41:26.416901 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406309    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:41:26.417531 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406312    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:41:26.417531 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406314    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:41:26.417531 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406316    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:41:26.417531 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406319    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:41:26.417531 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406322    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:41:26.417531 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406324    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:41:26.417531 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406326    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:41:26.417531 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406329    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:41:26.417531 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406332    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:41:26.417531 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406335    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:41:26.417531 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406338    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:41:26.417531 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406340    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:41:26.417531 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406343    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:41:26.417531 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406346    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:41:26.417531 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406348    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:41:26.417531 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406351    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:41:26.417531 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406353    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:41:26.417531 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406355    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:41:26.417531 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406358    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:41:26.417531 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406361    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:41:26.418241 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406363    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:41:26.418241 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406366    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:41:26.418241 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406368    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:41:26.418241 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406371    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:41:26.418241 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406373    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:41:26.418241 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406376    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:41:26.418241 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406378    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:41:26.418241 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406381    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:41:26.418241 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406383    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:41:26.418241 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406386    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:41:26.418241 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406389    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:41:26.418241 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406391    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:41:26.418241 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406393    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:41:26.418241 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406396    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:41:26.418241 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406398    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:41:26.418241 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406401    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:41:26.418241 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406403    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:41:26.418241 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406405    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:41:26.418241 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406408    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:41:26.418241 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406410    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:41:26.418937 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406413    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:41:26.418937 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406417    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:41:26.418937 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406419    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:41:26.418937 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406422    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:41:26.418937 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406424    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:41:26.418937 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406429    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:41:26.418937 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406433    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:41:26.418937 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406436    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:41:26.418937 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406440    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:41:26.418937 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406442    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:41:26.418937 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406445    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:41:26.418937 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406448    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:41:26.418937 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406450    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:41:26.418937 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.406474    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:41:26.418937 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406509    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:41:26.419310 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406513    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:41:26.419310 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406516    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:41:26.419310 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406519    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:41:26.419310 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406521    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:41:26.419310 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406524    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:41:26.419310 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406526    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:41:26.419310 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406529    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:41:26.419310 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406533    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:41:26.419310 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406535    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:41:26.419310 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406538    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:41:26.419310 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406541    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:41:26.419310 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406543    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:41:26.419310 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406546    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:41:26.419310 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406549    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:41:26.419310 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406551    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:41:26.419310 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406554    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:41:26.419310 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406557    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:41:26.419310 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406559    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:41:26.419310 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406563    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:41:26.419310 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406566    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:41:26.419781 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406568    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:41:26.419781 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406571    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:41:26.419781 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406573    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:41:26.419781 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406576    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:41:26.419781 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406579    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:41:26.419781 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406582    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:41:26.419781 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406585    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:41:26.419781 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406587    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:41:26.419781 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406590    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:41:26.419781 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406592    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:41:26.419781 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406594    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:41:26.419781 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406597    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:41:26.419781 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406599    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:41:26.419781 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406602    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:41:26.419781 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406604    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:41:26.419781 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406607    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:41:26.419781 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406609    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:41:26.419781 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406612    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:41:26.419781 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406615    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:41:26.419781 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406619    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:41:26.420249 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406621    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:41:26.420249 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406624    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:41:26.420249 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406627    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:41:26.420249 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406630    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:41:26.420249 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406633    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:41:26.420249 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406635    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:41:26.420249 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406638    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:41:26.420249 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406640    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:41:26.420249 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406643    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:41:26.420249 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406645    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:41:26.420249 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406649    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:41:26.420249 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406652    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:41:26.420249 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406655    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:41:26.420249 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406657    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:41:26.420249 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406660    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:41:26.420249 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406662    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:41:26.420249 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406666    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:41:26.420249 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406668    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:41:26.420249 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406671    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:41:26.420249 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406673    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:41:26.420704 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406676    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:41:26.420704 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406678    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:41:26.420704 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406681    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:41:26.420704 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406684    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:41:26.420704 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406688    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:41:26.420704 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406692    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:41:26.420704 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406695    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:41:26.420704 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406698    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:41:26.420704 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406701    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:41:26.420704 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406703    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:41:26.420704 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406706    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:41:26.420704 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406709    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:41:26.420704 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406712    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:41:26.420704 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406714    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:41:26.420704 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406717    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:41:26.420704 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406719    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:41:26.420704 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406722    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:41:26.420704 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406724    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:41:26.420704 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406727    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:41:26.421163 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406730    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:41:26.421163 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406732    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:41:26.421163 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406735    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:41:26.421163 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406738    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:41:26.421163 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406741    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:41:26.421163 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406744    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:41:26.421163 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406746    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:41:26.421163 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406749    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:41:26.421163 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406752    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:41:26.421163 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406754    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:41:26.421163 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406758    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:41:26.421163 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406760    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:41:26.421163 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406763    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:41:26.421163 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406765    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:41:26.421163 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406780    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:41:26.421163 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406783    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:41:26.421163 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406785    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:41:26.421163 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406788    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:41:26.421163 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406790    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:41:26.421163 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406793    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:41:26.421625 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406796    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:41:26.421625 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406799    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:41:26.421625 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406803    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:41:26.421625 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406807    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:41:26.421625 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406811    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:41:26.421625 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.406813    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:41:26.421625 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.414379    2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:41:26.421625 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.414396    2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:41:26.421625 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.414414    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:41:26.421625 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.414421    2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:41:26.421625 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414533    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:41:26.421625 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414541    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:41:26.421625 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414547    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:41:26.421625 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414552    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:41:26.421625 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414557    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:41:26.421625 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414561    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:41:26.421625 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414566    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:41:26.421625 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414570    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:41:26.421625 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414574    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:41:26.422102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414578    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:41:26.422102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414582    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:41:26.422102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414586    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:41:26.422102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414605    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:41:26.422102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414611    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:41:26.422102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414615    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:41:26.422102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414619    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:41:26.422102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414623    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:41:26.422102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414627    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:41:26.422102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414631    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:41:26.422102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414637    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:41:26.422102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414643    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:41:26.422102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414648    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:41:26.422102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414652    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:41:26.422102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414656    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:41:26.422102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414660    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:41:26.422102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414664    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:41:26.422102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414669    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:41:26.422102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414673    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:41:26.422539 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414679    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:41:26.422539 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414683    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:41:26.422539 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414688    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:41:26.422539 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414692    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:41:26.422539 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414696    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:41:26.422539 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414700    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:41:26.422539 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414704    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:41:26.422539 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414708    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:41:26.422539 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414713    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:41:26.422539 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414717    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:41:26.422539 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414721    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:41:26.422539 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414725    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:41:26.422539 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414729    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:41:26.422539 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414733    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:41:26.422539 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414737    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:41:26.422539 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414741    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:41:26.422539 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414745    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:41:26.422539 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414756    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:41:26.422539 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414761    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:41:26.422539 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414765    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:41:26.423144 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414792    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:41:26.423144 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414796    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:41:26.423144 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414801    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:41:26.423144 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414806    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:41:26.423144 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414810    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:41:26.423144 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414814    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:41:26.423144 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414818    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:41:26.423144 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414822    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:41:26.423144 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414826    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:41:26.423144 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414830    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:41:26.423144 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414834    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:41:26.423144 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414838    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:41:26.423144 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414843    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:41:26.423144 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414847    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:41:26.423144 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414851    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:41:26.423144 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414855    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:41:26.423144 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414859    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:41:26.423144 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414863    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:41:26.423144 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414867    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:41:26.423144 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414872    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:41:26.423601 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414876    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:41:26.423601 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414880    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:41:26.423601 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414884    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:41:26.423601 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414888    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:41:26.423601 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414892    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:41:26.423601 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414896    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:41:26.423601 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414900    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:41:26.423601 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414904    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:41:26.423601 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414911    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:41:26.423601 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414916    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:41:26.423601 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414928    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:41:26.423601 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414933    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:41:26.423601 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414937    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:41:26.423601 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414941    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:41:26.423601 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414945    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:41:26.423601 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414949    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:41:26.423601 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414954    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:41:26.423601 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414958    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:41:26.423601 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414962    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:41:26.423601 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414966    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:41:26.424093 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414971    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:41:26.424093 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414975    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:41:26.424093 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414979    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:41:26.424093 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414983    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:41:26.424093 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414988    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:41:26.424093 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414992    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:41:26.424093 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.414996    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:41:26.424093 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415001    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:41:26.424093 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415005    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:41:26.424093 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415009    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:41:26.424093 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415013    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:41:26.424093 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415017    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:41:26.424093 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415021    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:41:26.424093 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415025    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:41:26.424093 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415029    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:41:26.424093 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415033    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:41:26.424093 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415037    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:41:26.424093 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415041    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:41:26.424506 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.415081    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:41:26.424506 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415412    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:41:26.424506 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415421    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:41:26.424506 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415427    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:41:26.424506 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415431    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:41:26.424506 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415443    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:41:26.424506 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415448    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:41:26.424506 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415452    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:41:26.424506 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415456    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:41:26.424506 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415460    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:41:26.424506 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415465    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:41:26.424506 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415469    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:41:26.424506 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415473    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:41:26.424506 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415477    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:41:26.424506 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415481    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:41:26.424506 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415485    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:41:26.424893 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415489    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:41:26.424893 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415493    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:41:26.424893 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415497    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:41:26.424893 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415502    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:41:26.424893 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415506    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:41:26.424893 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415511    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:41:26.424893 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415517    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:41:26.424893 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415523    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:41:26.424893 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415528    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:41:26.424893 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415534    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:41:26.424893 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415539    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:41:26.424893 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415544    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:41:26.424893 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415549    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:41:26.424893 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415553    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:41:26.424893 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415557    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:41:26.424893 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415561    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:41:26.424893 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415565    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:41:26.424893 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415569    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:41:26.424893 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415574    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:41:26.425403 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415577    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:41:26.425403 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415582    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:41:26.425403 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415585    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:41:26.425403 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415597    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:41:26.425403 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415601    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:41:26.425403 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415605    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:41:26.425403 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415609    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:41:26.425403 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415613    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:41:26.425403 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415617    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:41:26.425403 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415621    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:41:26.425403 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415626    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:41:26.425403 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415630    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:41:26.425403 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415634    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:41:26.425403 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415637    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:41:26.425403 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415642    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:41:26.425403 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415646    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:41:26.425403 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415650    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:41:26.425403 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415654    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:41:26.425403 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415658    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:41:26.425403 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415662    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:41:26.425971 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415666    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:41:26.425971 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415670    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:41:26.425971 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415675    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:41:26.425971 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415679    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:41:26.425971 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415683    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:41:26.425971 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415687    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:41:26.425971 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415691    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:41:26.425971 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415695    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:41:26.425971 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415699    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:41:26.425971 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415703    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:41:26.425971 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415707    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:41:26.425971 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415713    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:41:26.425971 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415719    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:41:26.425971 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415724    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:41:26.425971 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415729    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:41:26.425971 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415734    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:41:26.425971 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415747    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:41:26.425971 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415752    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:41:26.425971 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415757    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:41:26.426436 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415761    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:41:26.426436 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415766    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:41:26.426436 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415788    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:41:26.426436 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415793    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:41:26.426436 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415798    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:41:26.426436 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415802    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:41:26.426436 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415806    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:41:26.426436 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415810    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:41:26.426436 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415814    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:41:26.426436 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415818    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:41:26.426436 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415822    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:41:26.426436 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415826    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:41:26.426436 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415830    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:41:26.426436 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415834    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:41:26.426436 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415838    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:41:26.426436 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415842    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:41:26.426436 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415847    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:41:26.426436 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415851    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:41:26.426436 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415854    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:41:26.426885 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415859    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:41:26.426885 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415863    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:41:26.426885 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415867    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:41:26.426885 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415871    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:41:26.426885 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415875    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:41:26.426885 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415879    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:41:26.426885 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415883    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:41:26.426885 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415887    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:41:26.426885 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415891    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:41:26.426885 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415895    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:41:26.426885 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415899    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:41:26.426885 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415913    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:41:26.426885 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415918    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:41:26.426885 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.415922    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:41:26.426885 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.415968    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:41:26.426885 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416038    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:41:26.427265 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416046    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:41:26.427265 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416051    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:41:26.427265 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416056    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:41:26.427265 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416060    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:41:26.427265 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416064    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:41:26.427265 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416068    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:41:26.427265 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416072    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:41:26.427265 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416076    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:41:26.427265 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416080    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:41:26.427265 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416084    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:41:26.427265 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416089    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:41:26.427265 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416093    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:41:26.427265 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416097    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:41:26.427265 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416102    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:41:26.427265 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416106    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:41:26.427265 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416109    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:41:26.427265 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416113    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:41:26.427265 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416117    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:41:26.427265 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416121    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:41:26.427265 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416126    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:41:26.427727 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416130    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:41:26.427727 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416141    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:41:26.427727 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416146    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:41:26.427727 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416150    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:41:26.427727 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416154    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:41:26.427727 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416158    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:41:26.427727 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416162    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:41:26.427727 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416165    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:41:26.427727 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416176    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:41:26.427727 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416180    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:41:26.427727 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416184    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:41:26.427727 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416188    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:41:26.427727 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416192    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:41:26.427727 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416196    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:41:26.427727 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416200    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:41:26.427727 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416204    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:41:26.427727 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416208    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:41:26.427727 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416212    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:41:26.427727 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416216    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:41:26.427727 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416220    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:41:26.428193 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416224    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:41:26.428193 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416228    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:41:26.428193 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416232    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:41:26.428193 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416236    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:41:26.428193 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416240    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:41:26.428193 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416244    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:41:26.428193 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416248    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:41:26.428193 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416252    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:41:26.428193 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416257    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:41:26.428193 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416263    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:41:26.428193 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416269    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:41:26.428193 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416274    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:41:26.428193 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416278    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:41:26.428193 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416283    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:41:26.428193 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416287    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:41:26.428193 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416291    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:41:26.428193 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416295    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:41:26.428193 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416299    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:41:26.428193 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416303    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:41:26.428647 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416307    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:41:26.428647 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416311    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:41:26.428647 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416323    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:41:26.428647 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416330    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:41:26.428647 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416334    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:41:26.428647 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416338    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:41:26.428647 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416342    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:41:26.428647 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416347    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:41:26.428647 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416351    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:41:26.428647 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416355    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:41:26.428647 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416359    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:41:26.428647 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416363    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:41:26.428647 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416367    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:41:26.428647 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416372    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:41:26.428647 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416377    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:41:26.428647 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416381    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:41:26.428647 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416385    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:41:26.428647 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416389    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:41:26.428647 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416394    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:41:26.428647 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416398    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:41:26.429102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416402    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:41:26.429102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416407    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:41:26.429102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416411    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:41:26.429102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416415    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:41:26.429102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416419    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:41:26.429102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416423    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:41:26.429102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416427    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:41:26.429102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416431    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:41:26.429102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416456    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:41:26.429102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416464    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:41:26.429102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416468    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:41:26.429102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416473    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:41:26.429102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416478    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:41:26.429102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416482    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:41:26.429102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416488    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:41:26.429102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416506    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:41:26.429102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416511    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:41:26.429102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416515    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:41:26.429102 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416520    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:41:26.429518 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416526    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:41:26.429518 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416530    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:41:26.429518 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416534    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:41:26.429518 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416538    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:41:26.429518 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416543    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:41:26.429518 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416547    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:41:26.429518 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:26.416551    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:41:26.429518 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.416625    2573 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:41:26.429518 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.420234    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 19:41:26.429518 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.420891    2573 server.go:1014] "Starting client certificate rotation"
Oct 30 19:41:26.429518 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.420983    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:41:26.429518 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.421583    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:41:26.429518 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.423063    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:41:26.429518 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.424202    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:41:26.435681 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.435667    2573 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:41:26.438245 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.438230    2573 log.go:25] "Validated CRI v1 image API"
Oct 30 19:41:26.439069 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.439047    2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:41:26.444517 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.444498    2573 fs.go:135] Filesystem UUIDs: map[163a5595-46e5-4348-8be0-1aa8714ab61b:/dev/nvme0n1p3 1dc77a52-92bf-417d-8811-05fd2f2384d5:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 19:41:26.444587 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.444516    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 19:41:26.450808 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.450648    2573 manager.go:217] Machine: {Timestamp:2026-10-30 19:41:26.448995938 +0000 UTC m=+0.303554660 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3187004 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f1a0197ff298c62c0f3881f4e119c SystemUUID:ec2f1a01-97ff-298c-62c0-f3881f4e119c BootID:64fdb448-a4e4-4170-a207-4dee29803448 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:31:2a:33:2a:8b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:31:2a:33:2a:8b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:f8:ab:cf:4c:58 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:41:26.450808 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.450803    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:41:26.450924 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.450908    2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:41:26.451794 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.451755    2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:41:26.451942 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.451795    2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-240.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:41:26.452380 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.452370    2573 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:41:26.452408 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.452383    2573 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:41:26.452433 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.452408    2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:41:26.452433 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.452420    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:41:26.453136 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.453126    2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:41:26.456528 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.456516    2573 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:41:26.456643 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.456634    2573 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:41:26.457149 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.457139    2573 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:41:26.457191 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.457161    2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:41:26.457191 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.457190    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:41:26.457268 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.457258    2573 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:41:26.457308 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.457288    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:41:26.458506 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.458482    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:41:26.461237 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.461216    2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:41:26.465972 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.465959    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:41:26.466816 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.466803    2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:41:26.466851 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.466834    2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:41:26.467013 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.467002    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:41:26.467052 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.467019    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:41:26.467052 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.467025    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:41:26.467052 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.467030    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:41:26.467052 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.467035    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:41:26.467052 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.467043    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:41:26.467662 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.467654    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:41:26.467690 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.467663    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:41:26.467690 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.467670    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:41:26.467690 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.467676    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:41:26.467690 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.467685    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:41:26.467888 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.467693    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:41:26.468299 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.468290    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:41:26.468327 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.468303    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:41:26.472038 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.472023    2573 server.go:1263] "Started kubelet"
Oct 30 19:41:26.472132 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.472090    2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:41:26.472223 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.472081    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:41:26.472276 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.472248    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:41:26.472691 ip-10-0-142-240 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:41:26.474375 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.474360    2573 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:41:26.476335 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.476314    2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:41:26.477283 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:26.477264    2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 19:41:26.479593 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.479577    2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:41:26.479593 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.479588    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:41:26.479758 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:26.479739    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-240.us-west-1.compute.internal\" not found"
Oct 30 19:41:26.480136 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.480119    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:41:26.480221 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.480132    2573 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:41:26.480221 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.480150    2573 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:41:26.480320 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.480280    2573 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:41:26.480320 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.480290    2573 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:41:26.480510 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.480482    2573 factory.go:55] Registering systemd factory
Oct 30 19:41:26.480510 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.480499    2573 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:41:26.480822 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.480805    2573 factory.go:153] Registering CRI-O factory
Oct 30 19:41:26.480867 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.480829    2573 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:41:26.480915 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.480874    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:41:26.480915 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.480887    2573 factory.go:103] Registering Raw factory
Oct 30 19:41:26.480915 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.480896    2573 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:41:26.481664 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.481650    2573 manager.go:319] Starting recovery of all containers
Oct 30 19:41:26.492005 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.491978    2573 manager.go:324] Recovery completed
Oct 30 19:41:26.496455 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.496438    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:41:26.498445 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.498423    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:41:26.498495 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.498460    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:41:26.498495 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.498470    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:41:26.498998 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.498985    2573 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:41:26.498998 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.498996    2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:41:26.499126 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.498990    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:41:26.499126 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.499011    2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:41:26.499668 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.499655    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:41:26.499732 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.499683    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:41:26.499732 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.499698    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:41:26.501132 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.501119    2573 policy_none.go:50] "Start"
Oct 30 19:41:26.501172 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.501136    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:41:26.501172 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.501145    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:41:26.503189 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.503179    2573 policy_none.go:44] "Start"
Oct 30 19:41:26.544199 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.544164    2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:41:26.546565 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.546553    2573 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:41:26.546952 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:26.546924    2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:41:26.546952 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.546951    2573 server.go:87] "Starting device plugin registration server"
Oct 30 19:41:26.547153 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.547140    2573 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:41:26.547201 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.547153    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:41:26.547361 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.547340    2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:41:26.547475 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.547430    2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:41:26.547475 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.547439    2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:41:26.547829 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:26.547813    2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:41:26.547902 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:26.547849    2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-240.us-west-1.compute.internal\" not found"
Oct 30 19:41:26.618216 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.618184    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:41:26.619437 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.619423    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:41:26.619517 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.619442    2573 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:41:26.619517 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.619459    2573 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:41:26.619517 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:26.619491    2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 19:41:26.624817 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.624800    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-240.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 19:41:26.625889 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:26.624801    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-240.us-west-1.compute.internal.18e3660de42538a3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-240.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-240.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 19:41:26.471841955 +0000 UTC m=+0.326400672,LastTimestamp:2026-10-30 19:41:26.471841955 +0000 UTC m=+0.326400672,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-240.us-west-1.compute.internal,}"
Oct 30 19:41:26.648168 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.648149    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:41:26.648859 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.648845    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:41:26.648926 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.648873    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:41:26.648926 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.648882    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:41:26.648926 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.648904    2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:41:26.673615 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:26.673562    2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-240.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 19:41:26.714447 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.714427    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:41:26.714539 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.714466    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kxmvn"
Oct 30 19:41:26.720482 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.720462    2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal"]
Oct 30 19:41:26.720532 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.720513    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:41:26.721369 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.721357    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:41:26.721437 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.721384    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:41:26.721437 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.721393    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:41:26.723368 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.723356    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:41:26.724169 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.724140    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:41:26.724271 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.724193    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:41:26.726033 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.726014    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:41:26.726122 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.726049    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:41:26.726122 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.726064    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:41:26.726122 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.726090    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:41:26.726122 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.726111    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:41:26.726122 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.726120    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:41:26.728028 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.728012    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:41:26.728094 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.728040    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:41:26.728701 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.728686    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:41:26.728765 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.728717    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:41:26.728765 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.728731    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:41:26.755224 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:26.755205    2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-240.us-west-1.compute.internal\" not found" node="ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:41:26.758426 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:26.758409    2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-240.us-west-1.compute.internal\" not found" node="ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:41:26.781016 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.780993    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce4e2caa0da71bc2cbc3d09da16fb366-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal\" (UID: \"ce4e2caa0da71bc2cbc3d09da16fb366\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:41:26.781092 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.781019    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84c50bb941905481e1c090101140d6a9-config\") pod \"kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal\" (UID: \"84c50bb941905481e1c090101140d6a9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:41:26.781092 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.781035    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ce4e2caa0da71bc2cbc3d09da16fb366-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal\" (UID: \"ce4e2caa0da71bc2cbc3d09da16fb366\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:41:26.786440 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.786425    2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:41:26.882066 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.882045    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ce4e2caa0da71bc2cbc3d09da16fb366-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal\" (UID: \"ce4e2caa0da71bc2cbc3d09da16fb366\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:41:26.882158 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.882068    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce4e2caa0da71bc2cbc3d09da16fb366-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal\" (UID: \"ce4e2caa0da71bc2cbc3d09da16fb366\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:41:26.882158 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.882085    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84c50bb941905481e1c090101140d6a9-config\") pod \"kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal\" (UID: \"84c50bb941905481e1c090101140d6a9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:41:26.882158 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.882135    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84c50bb941905481e1c090101140d6a9-config\") pod \"kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal\" (UID: \"84c50bb941905481e1c090101140d6a9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:41:26.882265 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.882156    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ce4e2caa0da71bc2cbc3d09da16fb366-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal\" (UID: \"ce4e2caa0da71bc2cbc3d09da16fb366\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:41:26.882265 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.882160    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce4e2caa0da71bc2cbc3d09da16fb366-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal\" (UID: \"ce4e2caa0da71bc2cbc3d09da16fb366\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:41:26.886741 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.886724    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:41:26.888062 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.888044    2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:41:26.888175 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.888158    2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:41:26.888689 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.888675    2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:41:26.888784 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:26.888695    2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-240.us-west-1.compute.internal\": node \"ip-10-0-142-240.us-west-1.compute.internal\" not found"
Oct 30 19:41:26.980791 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:26.980715    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:41:27.054844 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.054827    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:41:27.054925 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.054904    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:41:27.057980 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.057966    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:41:27.060815 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.060802    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:41:27.127691 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.127559    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:41:27.420723 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.420696    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 19:41:27.421291 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.420873    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 19:41:27.421291 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.420874    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 19:41:27.421291 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.420902    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 19:41:27.421291 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:27.420873    2573 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a8a25ac9f0dfc4e67bf96bf3051bc6f6-5b74f0dd39b75e91.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-142-240.us-west-1.compute.internal.18e3660de5bb7456\": read tcp 10.0.142.240:53708->3.13.225.239:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-142-240.us-west-1.compute.internal.18e3660de5bb7456  default   5561 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-240.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-240.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-240.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 19:41:26 +0000 UTC,LastTimestamp:2026-10-30 19:41:26.648877396 +0000 UTC m=+0.503436114,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-240.us-west-1.compute.internal,}"
Oct 30 19:41:27.458681 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.458661    2573 apiserver.go:52] "Watching apiserver"
Oct 30 19:41:27.479935 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.479913    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 19:41:27.718731 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.718630    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:36:28 +0000 UTC" deadline="2028-04-02 02:26:28.335898952 +0000 UTC"
Oct 30 19:41:27.718731 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.718676    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12462h45m0.617225784s"
Oct 30 19:41:27.747261 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.747237    2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:41:27.747560 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.747537    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-rwbsr","openshift-ovn-kubernetes/ovnkube-node-ft6pg","kube-system/kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-qz2rj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-snlg9","openshift-network-operator/iptables-alerter-dxw48","kube-system/global-pull-secret-syncer-8t48c","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52","openshift-image-registry/node-ca-9x594","openshift-multus/multus-l2mtk","openshift-multus/network-metrics-daemon-rj77q","openshift-dns/node-resolver-5qzfj","openshift-network-diagnostics/network-check-target-m9t6w"]
Oct 30 19:41:27.749949 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.749931    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rwbsr"
Oct 30 19:41:27.752012 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.751996    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.754079 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.754063    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.756140 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.756125    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:41:27.758226 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.758209    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dxw48"
Oct 30 19:41:27.760363 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.760343    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:27.760437 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:27.760411    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:41:27.762538 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.762517    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:41:27.765377 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.765364    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9x594"
Oct 30 19:41:27.767647 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.767633    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.769821 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.769809    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:27.769884 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:27.769868    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:41:27.772226 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.772210    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5qzfj"
Oct 30 19:41:27.774411 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.774396    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:41:27.774460 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:27.774447    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:41:27.781641 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.781627    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:41:27.788195 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788179    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs\") pod \"network-metrics-daemon-rj77q\" (UID: \"9cd48d14-cd97-4894-a43e-fa678a1833b3\") " pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:27.788243 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788204    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdwx2\" (UniqueName: \"kubernetes.io/projected/9b9838bc-8604-40f5-99d5-70c240eb439d-kube-api-access-cdwx2\") pod \"node-resolver-5qzfj\" (UID: \"9b9838bc-8604-40f5-99d5-70c240eb439d\") " pod="openshift-dns/node-resolver-5qzfj"
Oct 30 19:41:27.788243 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788222    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-node-log\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.788243 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788237    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/21e03037-74dd-44b2-bac7-791d30cf240d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:41:27.788413 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788253    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-multus-socket-dir-parent\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.788413 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788307    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-var-lib-cni-bin\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.788413 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788346    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c4d99d2-8ba6-410c-8337-f527e98d4651-multus-daemon-config\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.788413 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788384    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49wqq\" (UniqueName: \"kubernetes.io/projected/6391e2a6-bb94-4270-9fa9-f6d35f3e433c-kube-api-access-49wqq\") pod \"node-ca-9x594\" (UID: \"6391e2a6-bb94-4270-9fa9-f6d35f3e433c\") " pod="openshift-image-registry/node-ca-9x594"
Oct 30 19:41:27.788413 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788406    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-sysctl-d\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.788604 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788419    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-sys\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.788604 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788434    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/93cfc69d-646b-4910-972d-04c9076c6c3c-kubelet-config\") pod \"global-pull-secret-syncer-8t48c\" (UID: \"93cfc69d-646b-4910-972d-04c9076c6c3c\") " pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:27.788604 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788464    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-run-multus-certs\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.788604 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788486    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-sysconfig\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.788604 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788499    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-run\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.788604 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788536    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-tuned\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.788604 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788572    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/21e03037-74dd-44b2-bac7-791d30cf240d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:41:27.788604 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788603    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6391e2a6-bb94-4270-9fa9-f6d35f3e433c-serviceca\") pod \"node-ca-9x594\" (UID: \"6391e2a6-bb94-4270-9fa9-f6d35f3e433c\") " pod="openshift-image-registry/node-ca-9x594"
Oct 30 19:41:27.788965 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788638    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b9838bc-8604-40f5-99d5-70c240eb439d-tmp-dir\") pod \"node-resolver-5qzfj\" (UID: \"9b9838bc-8604-40f5-99d5-70c240eb439d\") " pod="openshift-dns/node-resolver-5qzfj"
Oct 30 19:41:27.788965 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788665    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-etc-openvswitch\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.788965 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788706    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-run-ovn-kubernetes\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.788965 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788733    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-log-socket\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.788965 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788748    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-cni-bin\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.788965 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788783    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-cni-netd\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.788965 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788812    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/023e1dd6-ffb9-48c8-a407-be580b23bddb-env-overrides\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.788965 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788850    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/023e1dd6-ffb9-48c8-a407-be580b23bddb-ovnkube-script-lib\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.788965 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788883    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-run-k8s-cni-cncf-io\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.788965 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788908    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-run-netns\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.788965 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788929    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-etc-kubernetes\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.788965 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788956    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-slash\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.789290 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.788980    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td8bh\" (UniqueName: \"kubernetes.io/projected/023e1dd6-ffb9-48c8-a407-be580b23bddb-kube-api-access-td8bh\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.789290 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789014    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-etc-selinux\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:41:27.789290 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789038    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2tsl\" (UniqueName: \"kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl\") pod \"network-check-target-m9t6w\" (UID: \"14e5d4c8-c53b-43b7-8301-804226ba1867\") " pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:41:27.789290 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789053    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-run-ovn\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.789290 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789084    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-sysctl-conf\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.789290 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789106    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7a30ded-2282-462e-bd34-dd4ad735a3ca-host-slash\") pod \"iptables-alerter-dxw48\" (UID: \"f7a30ded-2282-462e-bd34-dd4ad735a3ca\") " pod="openshift-network-operator/iptables-alerter-dxw48"
Oct 30 19:41:27.789290 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789126    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6391e2a6-bb94-4270-9fa9-f6d35f3e433c-host\") pod \"node-ca-9x594\" (UID: \"6391e2a6-bb94-4270-9fa9-f6d35f3e433c\") " pod="openshift-image-registry/node-ca-9x594"
Oct 30 19:41:27.789290 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789140    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.789290 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789168    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21e03037-74dd-44b2-bac7-791d30cf240d-system-cni-dir\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:41:27.789290 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789196    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-socket-dir\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:41:27.789290 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789216    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c4d99d2-8ba6-410c-8337-f527e98d4651-cni-binary-copy\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.789290 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789248    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-multus-conf-dir\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.789290 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789270    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-run-systemd\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.789290 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789283    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-cnibin\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.789660 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789330    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-var-lib-kubelet\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.789660 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789348    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-run-netns\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.789660 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789363    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/023e1dd6-ffb9-48c8-a407-be580b23bddb-ovn-node-metrics-cert\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.789660 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789411    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-var-lib-kubelet\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.789660 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789439    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk87g\" (UniqueName: \"kubernetes.io/projected/21e03037-74dd-44b2-bac7-791d30cf240d-kube-api-access-hk87g\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:41:27.789660 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789456    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-sys-fs\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:41:27.789660 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789484    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-systemd-units\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.789660 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789503    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g28pf\" (UniqueName: \"kubernetes.io/projected/043ccaed-b00b-4406-b156-eb95ea8e05f3-kube-api-access-g28pf\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.789660 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789517    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc48c\" (UniqueName: \"kubernetes.io/projected/1c4d99d2-8ba6-410c-8337-f527e98d4651-kube-api-access-rc48c\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.789660 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789534    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-kubernetes\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.789660 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789563    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cktlc\" (UniqueName: \"kubernetes.io/projected/f7a30ded-2282-462e-bd34-dd4ad735a3ca-kube-api-access-cktlc\") pod \"iptables-alerter-dxw48\" (UID: \"f7a30ded-2282-462e-bd34-dd4ad735a3ca\") " pod="openshift-network-operator/iptables-alerter-dxw48"
Oct 30 19:41:27.789660 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789587    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4xj4\" (UniqueName: \"kubernetes.io/projected/9cd48d14-cd97-4894-a43e-fa678a1833b3-kube-api-access-g4xj4\") pod \"network-metrics-daemon-rj77q\" (UID: \"9cd48d14-cd97-4894-a43e-fa678a1833b3\") " pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:27.789660 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789613    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ba9ed4cb-13c2-4ead-b549-5469e2724f6c-konnectivity-ca\") pod \"konnectivity-agent-rwbsr\" (UID: \"ba9ed4cb-13c2-4ead-b549-5469e2724f6c\") " pod="kube-system/konnectivity-agent-rwbsr"
Oct 30 19:41:27.789660 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789640    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-run-openvswitch\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.789660 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789663    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-modprobe-d\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.790068 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789679    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-device-dir\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:41:27.790068 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789697    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-multus-cni-dir\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.790068 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789716    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-system-cni-dir\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.790068 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789730    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ba9ed4cb-13c2-4ead-b549-5469e2724f6c-agent-certs\") pod \"konnectivity-agent-rwbsr\" (UID: \"ba9ed4cb-13c2-4ead-b549-5469e2724f6c\") " pod="kube-system/konnectivity-agent-rwbsr"
Oct 30 19:41:27.790068 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789759    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-kubelet\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.790068 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789786    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-host\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.790068 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789812    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21e03037-74dd-44b2-bac7-791d30cf240d-cni-binary-copy\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:41:27.790068 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789834    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21e03037-74dd-44b2-bac7-791d30cf240d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:41:27.790068 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789854    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-hostroot\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.790068 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789876    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f7a30ded-2282-462e-bd34-dd4ad735a3ca-iptables-alerter-script\") pod \"iptables-alerter-dxw48\" (UID: \"f7a30ded-2282-462e-bd34-dd4ad735a3ca\") " pod="openshift-network-operator/iptables-alerter-dxw48"
Oct 30 19:41:27.790068 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789895    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9b9838bc-8604-40f5-99d5-70c240eb439d-hosts-file\") pod \"node-resolver-5qzfj\" (UID: \"9b9838bc-8604-40f5-99d5-70c240eb439d\") " pod="openshift-dns/node-resolver-5qzfj"
Oct 30 19:41:27.790068 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789910    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/043ccaed-b00b-4406-b156-eb95ea8e05f3-tmp\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.790068 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789935    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/93cfc69d-646b-4910-972d-04c9076c6c3c-dbus\") pod \"global-pull-secret-syncer-8t48c\" (UID: \"93cfc69d-646b-4910-972d-04c9076c6c3c\") " pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:27.790068 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789955    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret\") pod \"global-pull-secret-syncer-8t48c\" (UID: \"93cfc69d-646b-4910-972d-04c9076c6c3c\") " pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:27.790068 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.789980    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:41:27.790068 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.790021    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5ngc\" (UniqueName: \"kubernetes.io/projected/08a234af-5c01-4901-80ae-2ec70fa08bfe-kube-api-access-g5ngc\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:41:27.790473 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.790043    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/023e1dd6-ffb9-48c8-a407-be580b23bddb-ovnkube-config\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.790473 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.790058    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-systemd\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.790473 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.790081    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-lib-modules\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.790473 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.790120    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-var-lib-openvswitch\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.790473 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.790135    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21e03037-74dd-44b2-bac7-791d30cf240d-cnibin\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:41:27.790473 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.790157    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-registration-dir\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:41:27.790473 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.790174    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-var-lib-cni-multus\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.799498 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.799480    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:41:27.799586 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.799547    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gbgr9"
Oct 30 19:41:27.831209 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.831137    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:41:27.831299 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.831232    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:41:27.831299 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.831236    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:41:27.831387 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.831312    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:41:27.831387 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.831316    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:41:27.831508 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.831492    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:41:27.831551 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.831515    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:41:27.831687 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.831670    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:41:27.831733 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.831690    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:41:27.831803 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.831729    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:41:27.831803 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.831762    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:41:27.832734 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.832718    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:41:27.833013 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.832998    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:41:27.833190 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.833171    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-shc6v\""
Oct 30 19:41:27.833299 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.833198    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hsfbm\""
Oct 30 19:41:27.833356 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.833299    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-5r6h9\""
Oct 30 19:41:27.833356 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.833301    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:41:27.855849 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.855833    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6dhxv\""
Oct 30 19:41:27.855849 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.855837    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-47f99\""
Oct 30 19:41:27.856361 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.856346    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:41:27.856526 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.856513    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:41:27.856619 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.856608    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:41:27.858163 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.858149    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:41:27.858236 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.858214    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pjk7j\""
Oct 30 19:41:27.880032 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.880014    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pcf9z\""
Oct 30 19:41:27.880813 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.880763    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tqb6z\""
Oct 30 19:41:27.880813 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.880808    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:41:27.880950 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.880818    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9bk6w\""
Oct 30 19:41:27.881225 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.881158    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:41:27.881225 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.881164    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:41:27.881225 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.881182    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:41:27.881225 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.881199    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:41:27.881225 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.881215    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:41:27.881225 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.881169    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:41:27.881441 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.881182    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:41:27.881441 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.881199    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:41:27.890705 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.890689    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-kubernetes\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.890811 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.890713    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cktlc\" (UniqueName: \"kubernetes.io/projected/f7a30ded-2282-462e-bd34-dd4ad735a3ca-kube-api-access-cktlc\") pod \"iptables-alerter-dxw48\" (UID: \"f7a30ded-2282-462e-bd34-dd4ad735a3ca\") " pod="openshift-network-operator/iptables-alerter-dxw48"
Oct 30 19:41:27.890811 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.890729    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4xj4\" (UniqueName: \"kubernetes.io/projected/9cd48d14-cd97-4894-a43e-fa678a1833b3-kube-api-access-g4xj4\") pod \"network-metrics-daemon-rj77q\" (UID: \"9cd48d14-cd97-4894-a43e-fa678a1833b3\") " pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:27.890811 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.890744    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ba9ed4cb-13c2-4ead-b549-5469e2724f6c-konnectivity-ca\") pod \"konnectivity-agent-rwbsr\" (UID: \"ba9ed4cb-13c2-4ead-b549-5469e2724f6c\") " pod="kube-system/konnectivity-agent-rwbsr"
Oct 30 19:41:27.890962 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.890837    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-kubernetes\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.890962 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.890885    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-run-openvswitch\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.890962 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.890924    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-modprobe-d\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.890962 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.890949    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-device-dir\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:41:27.891123 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891003    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-device-dir\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:41:27.891123 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891007    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-run-openvswitch\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.891123 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891031    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-multus-cni-dir\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.891123 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891062    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-modprobe-d\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.891123 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891076    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-system-cni-dir\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.891123 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891112    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ba9ed4cb-13c2-4ead-b549-5469e2724f6c-agent-certs\") pod \"konnectivity-agent-rwbsr\" (UID: \"ba9ed4cb-13c2-4ead-b549-5469e2724f6c\") " pod="kube-system/konnectivity-agent-rwbsr"
Oct 30 19:41:27.891387 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891135    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-kubelet\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.891387 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891166    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-host\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.891387 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891172    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-multus-cni-dir\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.891387 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891191    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21e03037-74dd-44b2-bac7-791d30cf240d-cni-binary-copy\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:41:27.891387 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891193    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-kubelet\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.891387 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891180    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-system-cni-dir\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.891387 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891216    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21e03037-74dd-44b2-bac7-791d30cf240d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:41:27.891387 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891230    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-host\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.891387 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891263    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ba9ed4cb-13c2-4ead-b549-5469e2724f6c-konnectivity-ca\") pod \"konnectivity-agent-rwbsr\" (UID: \"ba9ed4cb-13c2-4ead-b549-5469e2724f6c\") " pod="kube-system/konnectivity-agent-rwbsr"
Oct 30 19:41:27.891387 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891276    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-hostroot\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.891387 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891305    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f7a30ded-2282-462e-bd34-dd4ad735a3ca-iptables-alerter-script\") pod \"iptables-alerter-dxw48\" (UID: \"f7a30ded-2282-462e-bd34-dd4ad735a3ca\") " pod="openshift-network-operator/iptables-alerter-dxw48"
Oct 30 19:41:27.891387 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891333    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9b9838bc-8604-40f5-99d5-70c240eb439d-hosts-file\") pod \"node-resolver-5qzfj\" (UID: \"9b9838bc-8604-40f5-99d5-70c240eb439d\") " pod="openshift-dns/node-resolver-5qzfj"
Oct 30 19:41:27.891387 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891356    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/043ccaed-b00b-4406-b156-eb95ea8e05f3-tmp\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.891387 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891375    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21e03037-74dd-44b2-bac7-791d30cf240d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:41:27.891998 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891418    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9b9838bc-8604-40f5-99d5-70c240eb439d-hosts-file\") pod \"node-resolver-5qzfj\" (UID: \"9b9838bc-8604-40f5-99d5-70c240eb439d\") " pod="openshift-dns/node-resolver-5qzfj"
Oct 30 19:41:27.891998 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891428    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/93cfc69d-646b-4910-972d-04c9076c6c3c-dbus\") pod \"global-pull-secret-syncer-8t48c\" (UID: \"93cfc69d-646b-4910-972d-04c9076c6c3c\") " pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:27.891998 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891426    2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:41:27.891998 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891468    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-hostroot\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.891998 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891378    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/93cfc69d-646b-4910-972d-04c9076c6c3c-dbus\") pod \"global-pull-secret-syncer-8t48c\" (UID: \"93cfc69d-646b-4910-972d-04c9076c6c3c\") " pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:27.891998 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891501    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret\") pod \"global-pull-secret-syncer-8t48c\" (UID: \"93cfc69d-646b-4910-972d-04c9076c6c3c\") " pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:27.891998 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891525    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:41:27.891998 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891552    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5ngc\" (UniqueName: \"kubernetes.io/projected/08a234af-5c01-4901-80ae-2ec70fa08bfe-kube-api-access-g5ngc\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:41:27.891998 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891573    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/023e1dd6-ffb9-48c8-a407-be580b23bddb-ovnkube-config\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.891998 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891595    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-systemd\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.891998 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891618    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-lib-modules\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.891998 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891643    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-var-lib-openvswitch\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.891998 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891665    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21e03037-74dd-44b2-bac7-791d30cf240d-cnibin\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:41:27.891998 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891688    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-registration-dir\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:41:27.891998 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891711    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-var-lib-cni-multus\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.891998 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891734    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs\") pod \"network-metrics-daemon-rj77q\" (UID: \"9cd48d14-cd97-4894-a43e-fa678a1833b3\") " pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:27.891998 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891761    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdwx2\" (UniqueName: \"kubernetes.io/projected/9b9838bc-8604-40f5-99d5-70c240eb439d-kube-api-access-cdwx2\") pod \"node-resolver-5qzfj\" (UID: \"9b9838bc-8604-40f5-99d5-70c240eb439d\") " pod="openshift-dns/node-resolver-5qzfj"
Oct 30 19:41:27.892924 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891819    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21e03037-74dd-44b2-bac7-791d30cf240d-cni-binary-copy\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:41:27.892924 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891828    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-node-log\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.892924 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891858    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/21e03037-74dd-44b2-bac7-791d30cf240d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:41:27.892924 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891884    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-multus-socket-dir-parent\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.892924 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891907    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-var-lib-cni-bin\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.892924 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891951    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c4d99d2-8ba6-410c-8337-f527e98d4651-multus-daemon-config\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.892924 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891964    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f7a30ded-2282-462e-bd34-dd4ad735a3ca-iptables-alerter-script\") pod \"iptables-alerter-dxw48\" (UID: \"f7a30ded-2282-462e-bd34-dd4ad735a3ca\") " pod="openshift-network-operator/iptables-alerter-dxw48"
Oct 30 19:41:27.892924 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891981    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49wqq\" (UniqueName: \"kubernetes.io/projected/6391e2a6-bb94-4270-9fa9-f6d35f3e433c-kube-api-access-49wqq\") pod \"node-ca-9x594\" (UID: \"6391e2a6-bb94-4270-9fa9-f6d35f3e433c\") " pod="openshift-image-registry/node-ca-9x594"
Oct 30 19:41:27.892924 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:27.891993    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:41:27.892924 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.892009    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-sysctl-d\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.892924 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.892026    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-var-lib-openvswitch\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.892924 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.892031    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-sys\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.892924 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.892071    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21e03037-74dd-44b2-bac7-791d30cf240d-cnibin\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:41:27.892924 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.892076    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-node-log\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.892924 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:27.892075    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret podName:93cfc69d-646b-4910-972d-04c9076c6c3c nodeName:}" failed. No retries permitted until 2026-10-30 19:41:28.392041218 +0000 UTC m=+2.246599926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret") pod "global-pull-secret-syncer-8t48c" (UID: "93cfc69d-646b-4910-972d-04c9076c6c3c") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:41:27.892924 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.892112    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:41:27.892924 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.892117    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-registration-dir\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:41:27.893676 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.892161    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-var-lib-cni-multus\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.893676 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.892176    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-systemd\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.893676 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.892499    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/21e03037-74dd-44b2-bac7-791d30cf240d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:41:27.893676 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.892664    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/023e1dd6-ffb9-48c8-a407-be580b23bddb-ovnkube-config\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.893676 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:27.892747    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:41:27.893676 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.892837    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-sys\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.893676 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.892838    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c4d99d2-8ba6-410c-8337-f527e98d4651-multus-daemon-config\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.893676 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.892913    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-multus-socket-dir-parent\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.893676 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.892963    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-var-lib-cni-bin\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.893676 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.891926    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-lib-modules\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.893676 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.892993    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-sysctl-d\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.893676 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:27.893036    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs podName:9cd48d14-cd97-4894-a43e-fa678a1833b3 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:28.393022444 +0000 UTC m=+2.247581154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs") pod "network-metrics-daemon-rj77q" (UID: "9cd48d14-cd97-4894-a43e-fa678a1833b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:41:27.893676 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893072    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/93cfc69d-646b-4910-972d-04c9076c6c3c-kubelet-config\") pod \"global-pull-secret-syncer-8t48c\" (UID: \"93cfc69d-646b-4910-972d-04c9076c6c3c\") " pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:27.893676 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893092    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-run-multus-certs\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.893676 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893112    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-sysconfig\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.893676 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893132    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-run\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.893676 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893164    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-run-multus-certs\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.894437 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893169    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-tuned\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.894437 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893182    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-sysconfig\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.894437 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893205    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/21e03037-74dd-44b2-bac7-791d30cf240d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:41:27.894437 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893213    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-run\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.894437 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893222    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/93cfc69d-646b-4910-972d-04c9076c6c3c-kubelet-config\") pod \"global-pull-secret-syncer-8t48c\" (UID: \"93cfc69d-646b-4910-972d-04c9076c6c3c\") " pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:27.894437 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893232    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6391e2a6-bb94-4270-9fa9-f6d35f3e433c-serviceca\") pod \"node-ca-9x594\" (UID: \"6391e2a6-bb94-4270-9fa9-f6d35f3e433c\") " pod="openshift-image-registry/node-ca-9x594"
Oct 30 19:41:27.894437 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893255    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b9838bc-8604-40f5-99d5-70c240eb439d-tmp-dir\") pod \"node-resolver-5qzfj\" (UID: \"9b9838bc-8604-40f5-99d5-70c240eb439d\") " pod="openshift-dns/node-resolver-5qzfj"
Oct 30 19:41:27.894437 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893279    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-etc-openvswitch\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.894437 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893302    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-run-ovn-kubernetes\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.894437 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893327    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-log-socket\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.894437 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893360    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-cni-bin\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.894437 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893384    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-cni-netd\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.894437 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893405    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/023e1dd6-ffb9-48c8-a407-be580b23bddb-env-overrides\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.894437 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893435    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/023e1dd6-ffb9-48c8-a407-be580b23bddb-ovnkube-script-lib\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.894437 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893462    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-run-k8s-cni-cncf-io\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.894437 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893484    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-run-netns\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.894437 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893508    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-etc-kubernetes\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.894437 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893531    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-slash\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.895079 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893574    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td8bh\" (UniqueName: \"kubernetes.io/projected/023e1dd6-ffb9-48c8-a407-be580b23bddb-kube-api-access-td8bh\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.895079 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893599    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-etc-selinux\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:41:27.895079 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893632    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2tsl\" (UniqueName: \"kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl\") pod \"network-check-target-m9t6w\" (UID: \"14e5d4c8-c53b-43b7-8301-804226ba1867\") " pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:41:27.895079 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893661    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6391e2a6-bb94-4270-9fa9-f6d35f3e433c-serviceca\") pod \"node-ca-9x594\" (UID: \"6391e2a6-bb94-4270-9fa9-f6d35f3e433c\") " pod="openshift-image-registry/node-ca-9x594"
Oct 30 19:41:27.895079 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893688    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/21e03037-74dd-44b2-bac7-791d30cf240d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:41:27.895079 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893696    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-run-ovn\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.895079 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893749    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-slash\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.895079 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893802    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-run-k8s-cni-cncf-io\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.895079 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893811    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-log-socket\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.895079 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893843    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-run-netns\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.895079 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893854    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-sysctl-conf\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.895079 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893872    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-etc-openvswitch\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.895079 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893879    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7a30ded-2282-462e-bd34-dd4ad735a3ca-host-slash\") pod \"iptables-alerter-dxw48\" (UID: \"f7a30ded-2282-462e-bd34-dd4ad735a3ca\") " pod="openshift-network-operator/iptables-alerter-dxw48"
Oct 30 19:41:27.895079 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893883    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-etc-kubernetes\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.895079 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893910    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-cni-netd\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.895079 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893916    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-run-ovn-kubernetes\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.895079 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893941    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6391e2a6-bb94-4270-9fa9-f6d35f3e433c-host\") pod \"node-ca-9x594\" (UID: \"6391e2a6-bb94-4270-9fa9-f6d35f3e433c\") " pod="openshift-image-registry/node-ca-9x594"
Oct 30 19:41:27.895653 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893967    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.895653 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894003    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-etc-selinux\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:41:27.895653 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894014    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21e03037-74dd-44b2-bac7-791d30cf240d-system-cni-dir\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:41:27.895653 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894040    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-socket-dir\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:41:27.895653 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894061    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c4d99d2-8ba6-410c-8337-f527e98d4651-cni-binary-copy\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.895653 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894085    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-multus-conf-dir\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.895653 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894095    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/023e1dd6-ffb9-48c8-a407-be580b23bddb-ovnkube-script-lib\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.895653 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894109    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-run-systemd\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.895653 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894133    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-cnibin\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.895653 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894156    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-var-lib-kubelet\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.895653 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894181    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-run-netns\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.895653 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894228    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/023e1dd6-ffb9-48c8-a407-be580b23bddb-ovn-node-metrics-cert\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.895653 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894242    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-run-ovn\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.895653 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894255    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-var-lib-kubelet\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.895653 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894282    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk87g\" (UniqueName: \"kubernetes.io/projected/21e03037-74dd-44b2-bac7-791d30cf240d-kube-api-access-hk87g\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:41:27.895653 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894306    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-sys-fs\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:41:27.895653 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894331    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-systemd-units\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.896124 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894350    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6391e2a6-bb94-4270-9fa9-f6d35f3e433c-host\") pod \"node-ca-9x594\" (UID: \"6391e2a6-bb94-4270-9fa9-f6d35f3e433c\") " pod="openshift-image-registry/node-ca-9x594"
Oct 30 19:41:27.896124 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894354    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g28pf\" (UniqueName: \"kubernetes.io/projected/043ccaed-b00b-4406-b156-eb95ea8e05f3-kube-api-access-g28pf\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.896124 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893941    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-cni-bin\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.896124 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894390    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7a30ded-2282-462e-bd34-dd4ad735a3ca-host-slash\") pod \"iptables-alerter-dxw48\" (UID: \"f7a30ded-2282-462e-bd34-dd4ad735a3ca\") " pod="openshift-network-operator/iptables-alerter-dxw48"
Oct 30 19:41:27.896124 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894393    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc48c\" (UniqueName: \"kubernetes.io/projected/1c4d99d2-8ba6-410c-8337-f527e98d4651-kube-api-access-rc48c\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.896124 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894408    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21e03037-74dd-44b2-bac7-791d30cf240d-system-cni-dir\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:41:27.896124 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894584    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.896124 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894734    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/023e1dd6-ffb9-48c8-a407-be580b23bddb-env-overrides\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.896124 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894751    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-socket-dir\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:41:27.896124 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894838    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-var-lib-kubelet\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.896124 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894840    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-var-lib-kubelet\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.896124 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.893734    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b9838bc-8604-40f5-99d5-70c240eb439d-tmp-dir\") pod \"node-resolver-5qzfj\" (UID: \"9b9838bc-8604-40f5-99d5-70c240eb439d\") " pod="openshift-dns/node-resolver-5qzfj"
Oct 30 19:41:27.896124 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894935    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-cnibin\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.896124 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894977    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-multus-conf-dir\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.896124 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.895020    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-systemd-units\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.896124 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.894357    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-sysctl-conf\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.896124 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.895074    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-sys-fs\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:41:27.896124 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.895096    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-run-systemd\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.896580 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.895138    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-run-netns\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.896580 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.895313    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/043ccaed-b00b-4406-b156-eb95ea8e05f3-tmp\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.896580 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.895475    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c4d99d2-8ba6-410c-8337-f527e98d4651-cni-binary-copy\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.896580 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.895490    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-tuned\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.896580 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.895670    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ba9ed4cb-13c2-4ead-b549-5469e2724f6c-agent-certs\") pod \"konnectivity-agent-rwbsr\" (UID: \"ba9ed4cb-13c2-4ead-b549-5469e2724f6c\") " pod="kube-system/konnectivity-agent-rwbsr"
Oct 30 19:41:27.897051 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.897035    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/023e1dd6-ffb9-48c8-a407-be580b23bddb-ovn-node-metrics-cert\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.964303 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.964273    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5ngc\" (UniqueName: \"kubernetes.io/projected/08a234af-5c01-4901-80ae-2ec70fa08bfe-kube-api-access-g5ngc\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:41:27.965230 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.965205    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td8bh\" (UniqueName: \"kubernetes.io/projected/023e1dd6-ffb9-48c8-a407-be580b23bddb-kube-api-access-td8bh\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:27.965547 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.965527    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g28pf\" (UniqueName: \"kubernetes.io/projected/043ccaed-b00b-4406-b156-eb95ea8e05f3-kube-api-access-g28pf\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:27.969127 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.969078    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc48c\" (UniqueName: \"kubernetes.io/projected/1c4d99d2-8ba6-410c-8337-f527e98d4651-kube-api-access-rc48c\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:27.969950 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.969934    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk87g\" (UniqueName: \"kubernetes.io/projected/21e03037-74dd-44b2-bac7-791d30cf240d-kube-api-access-hk87g\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:41:27.978843 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.978826    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cktlc\" (UniqueName: \"kubernetes.io/projected/f7a30ded-2282-462e-bd34-dd4ad735a3ca-kube-api-access-cktlc\") pod \"iptables-alerter-dxw48\" (UID: \"f7a30ded-2282-462e-bd34-dd4ad735a3ca\") " pod="openshift-network-operator/iptables-alerter-dxw48"
Oct 30 19:41:27.980027 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:27.980013    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:41:27.980083 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:27.980030    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:41:27.980083 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:27.980041    2573 projected.go:196] Error preparing data for projected volume kube-api-access-v2tsl for pod openshift-network-diagnostics/network-check-target-m9t6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:41:27.980140 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:27.980087    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl podName:14e5d4c8-c53b-43b7-8301-804226ba1867 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:28.48007427 +0000 UTC m=+2.334632989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-v2tsl" (UniqueName: "kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl") pod "network-check-target-m9t6w" (UID: "14e5d4c8-c53b-43b7-8301-804226ba1867") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:41:27.981971 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.981952    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49wqq\" (UniqueName: \"kubernetes.io/projected/6391e2a6-bb94-4270-9fa9-f6d35f3e433c-kube-api-access-49wqq\") pod \"node-ca-9x594\" (UID: \"6391e2a6-bb94-4270-9fa9-f6d35f3e433c\") " pod="openshift-image-registry/node-ca-9x594"
Oct 30 19:41:27.983388 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.983374    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdwx2\" (UniqueName: \"kubernetes.io/projected/9b9838bc-8604-40f5-99d5-70c240eb439d-kube-api-access-cdwx2\") pod \"node-resolver-5qzfj\" (UID: \"9b9838bc-8604-40f5-99d5-70c240eb439d\") " pod="openshift-dns/node-resolver-5qzfj"
Oct 30 19:41:27.986013 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:27.985998    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4xj4\" (UniqueName: \"kubernetes.io/projected/9cd48d14-cd97-4894-a43e-fa678a1833b3-kube-api-access-g4xj4\") pod \"network-metrics-daemon-rj77q\" (UID: \"9cd48d14-cd97-4894-a43e-fa678a1833b3\") " pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:28.070535 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:28.070509    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rwbsr"
Oct 30 19:41:28.114638 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:28.114598    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:28.117664 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:28.117643    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:41:28.143811 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:28.143783    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:41:28.143920 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:28.143822    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dxw48"
Oct 30 19:41:28.170180 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:28.170158    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:41:28.182658 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:28.182617    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9x594"
Oct 30 19:41:28.200254 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:28.200224    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6391e2a6_bb94_4270_9fa9_f6d35f3e433c.slice/crio-e3af04fe42901cdc0b7120ac70d0486cdfdc32e2ecde57614459b61e6cbd3890 WatchSource:0}: Error finding container e3af04fe42901cdc0b7120ac70d0486cdfdc32e2ecde57614459b61e6cbd3890: Status 404 returned error can't find the container with id e3af04fe42901cdc0b7120ac70d0486cdfdc32e2ecde57614459b61e6cbd3890
Oct 30 19:41:28.200505 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:28.200447    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c50bb941905481e1c090101140d6a9.slice/crio-f309b7066d4709e2196330d74d66737ff20a52fc1f3bc1500bcaf68b20c4d8fd WatchSource:0}: Error finding container f309b7066d4709e2196330d74d66737ff20a52fc1f3bc1500bcaf68b20c4d8fd: Status 404 returned error can't find the container with id f309b7066d4709e2196330d74d66737ff20a52fc1f3bc1500bcaf68b20c4d8fd
Oct 30 19:41:28.201320 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:28.201300    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a234af_5c01_4901_80ae_2ec70fa08bfe.slice/crio-a5c4f6e7e4882cdb016ccfefe987c7727973340cee27ecb103b4cbad9ee2ffdc WatchSource:0}: Error finding container a5c4f6e7e4882cdb016ccfefe987c7727973340cee27ecb103b4cbad9ee2ffdc: Status 404 returned error can't find the container with id a5c4f6e7e4882cdb016ccfefe987c7727973340cee27ecb103b4cbad9ee2ffdc
Oct 30 19:41:28.201949 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:28.201931    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce4e2caa0da71bc2cbc3d09da16fb366.slice/crio-335c8f7981f2295a9fdf43c4ac98692985725068e925efbb0c5ad065f24102fa WatchSource:0}: Error finding container 335c8f7981f2295a9fdf43c4ac98692985725068e925efbb0c5ad065f24102fa: Status 404 returned error can't find the container with id 335c8f7981f2295a9fdf43c4ac98692985725068e925efbb0c5ad065f24102fa
Oct 30 19:41:28.204888 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:28.204873    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:41:28.208239 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:28.208222    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l2mtk"
Oct 30 19:41:28.211361 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:28.211340    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5qzfj"
Oct 30 19:41:28.217086 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:28.217069    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4d99d2_8ba6_410c_8337_f527e98d4651.slice/crio-7dc893beb226092892877d0b12c7ed7d67e66b92df4f82532a9ee892247a45eb WatchSource:0}: Error finding container 7dc893beb226092892877d0b12c7ed7d67e66b92df4f82532a9ee892247a45eb: Status 404 returned error can't find the container with id 7dc893beb226092892877d0b12c7ed7d67e66b92df4f82532a9ee892247a45eb
Oct 30 19:41:28.219069 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:28.219019    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9838bc_8604_40f5_99d5_70c240eb439d.slice/crio-b95c2d037efc75b7bec069c16ffe43df706d099dfab64de8e67b4fdd1e0f04dc WatchSource:0}: Error finding container b95c2d037efc75b7bec069c16ffe43df706d099dfab64de8e67b4fdd1e0f04dc: Status 404 returned error can't find the container with id b95c2d037efc75b7bec069c16ffe43df706d099dfab64de8e67b4fdd1e0f04dc
Oct 30 19:41:28.399145 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:28.399121    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret\") pod \"global-pull-secret-syncer-8t48c\" (UID: \"93cfc69d-646b-4910-972d-04c9076c6c3c\") " pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:28.399289 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:28.399153    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs\") pod \"network-metrics-daemon-rj77q\" (UID: \"9cd48d14-cd97-4894-a43e-fa678a1833b3\") " pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:28.399289 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:28.399269    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:41:28.399289 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:28.399269    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:41:28.399391 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:28.399330    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs podName:9cd48d14-cd97-4894-a43e-fa678a1833b3 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:29.399313552 +0000 UTC m=+3.253872257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs") pod "network-metrics-daemon-rj77q" (UID: "9cd48d14-cd97-4894-a43e-fa678a1833b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:41:28.399391 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:28.399342    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret podName:93cfc69d-646b-4910-972d-04c9076c6c3c nodeName:}" failed. No retries permitted until 2026-10-30 19:41:29.399336766 +0000 UTC m=+3.253895471 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret") pod "global-pull-secret-syncer-8t48c" (UID: "93cfc69d-646b-4910-972d-04c9076c6c3c") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:41:28.500193 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:28.500108    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2tsl\" (UniqueName: \"kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl\") pod \"network-check-target-m9t6w\" (UID: \"14e5d4c8-c53b-43b7-8301-804226ba1867\") " pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:41:28.500821 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:28.500249    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:41:28.500821 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:28.500266    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:41:28.500821 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:28.500277    2573 projected.go:196] Error preparing data for projected volume kube-api-access-v2tsl for pod openshift-network-diagnostics/network-check-target-m9t6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:41:28.500821 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:28.500331    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl podName:14e5d4c8-c53b-43b7-8301-804226ba1867 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:29.500316064 +0000 UTC m=+3.354874769 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2tsl" (UniqueName: "kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl") pod "network-check-target-m9t6w" (UID: "14e5d4c8-c53b-43b7-8301-804226ba1867") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:41:30.133649 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:30.133587    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5qzfj" event={"ID":"9b9838bc-8604-40f5-99d5-70c240eb439d","Type":"ContainerStarted","Data":"b95c2d037efc75b7bec069c16ffe43df706d099dfab64de8e67b4fdd1e0f04dc"}
Oct 30 19:41:30.134596 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:30.134571    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l2mtk" event={"ID":"1c4d99d2-8ba6-410c-8337-f527e98d4651","Type":"ContainerStarted","Data":"7dc893beb226092892877d0b12c7ed7d67e66b92df4f82532a9ee892247a45eb"}
Oct 30 19:41:30.135588 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:30.135568    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52" event={"ID":"08a234af-5c01-4901-80ae-2ec70fa08bfe","Type":"ContainerStarted","Data":"a5c4f6e7e4882cdb016ccfefe987c7727973340cee27ecb103b4cbad9ee2ffdc"}
Oct 30 19:41:30.136552 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:30.136537    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9x594" event={"ID":"6391e2a6-bb94-4270-9fa9-f6d35f3e433c","Type":"ContainerStarted","Data":"e3af04fe42901cdc0b7120ac70d0486cdfdc32e2ecde57614459b61e6cbd3890"}
Oct 30 19:41:30.137463 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:30.137433    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal" event={"ID":"84c50bb941905481e1c090101140d6a9","Type":"ContainerStarted","Data":"f309b7066d4709e2196330d74d66737ff20a52fc1f3bc1500bcaf68b20c4d8fd"}
Oct 30 19:41:30.138344 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:30.138327    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal" event={"ID":"ce4e2caa0da71bc2cbc3d09da16fb366","Type":"ContainerStarted","Data":"335c8f7981f2295a9fdf43c4ac98692985725068e925efbb0c5ad065f24102fa"}
Oct 30 19:41:30.312266 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:30.312176    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:36:29 +0000 UTC" deadline="2028-03-28 06:57:49.869968368 +0000 UTC"
Oct 30 19:41:30.312266 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:30.312215    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12347h16m19.5577559s"
Oct 30 19:41:30.320872 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:30.320854    2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:41:30.445511 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:30.445487    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod043ccaed_b00b_4406_b156_eb95ea8e05f3.slice/crio-c230cebc2ac527984ae7be71fd32fe26330fc871ccc702858134fb1848cf8146 WatchSource:0}: Error finding container c230cebc2ac527984ae7be71fd32fe26330fc871ccc702858134fb1848cf8146: Status 404 returned error can't find the container with id c230cebc2ac527984ae7be71fd32fe26330fc871ccc702858134fb1848cf8146
Oct 30 19:41:30.503732 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:30.503697    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023e1dd6_ffb9_48c8_a407_be580b23bddb.slice/crio-b31aeda5cb15fecf77cf84227228897b19991c47a8ee13fdea3b6aa2bcde28a7 WatchSource:0}: Error finding container b31aeda5cb15fecf77cf84227228897b19991c47a8ee13fdea3b6aa2bcde28a7: Status 404 returned error can't find the container with id b31aeda5cb15fecf77cf84227228897b19991c47a8ee13fdea3b6aa2bcde28a7
Oct 30 19:41:30.512124 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:30.512102    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:41:30.529607 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:30.529587    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a30ded_2282_462e_bd34_dd4ad735a3ca.slice/crio-96ea2175c027fdae0036df0774cf8de40dc933e8677e3cf7af9de89078144182 WatchSource:0}: Error finding container 96ea2175c027fdae0036df0774cf8de40dc933e8677e3cf7af9de89078144182: Status 404 returned error can't find the container with id 96ea2175c027fdae0036df0774cf8de40dc933e8677e3cf7af9de89078144182
Oct 30 19:41:30.539149 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:30.539123    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e03037_74dd_44b2_bac7_791d30cf240d.slice/crio-220752f888ce8a936d7ee2a4a0ec9c6b58882fb3286025b389130741ffc82a15 WatchSource:0}: Error finding container 220752f888ce8a936d7ee2a4a0ec9c6b58882fb3286025b389130741ffc82a15: Status 404 returned error can't find the container with id 220752f888ce8a936d7ee2a4a0ec9c6b58882fb3286025b389130741ffc82a15
Oct 30 19:41:30.558809 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:41:30.558775    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba9ed4cb_13c2_4ead_b549_5469e2724f6c.slice/crio-f89e87a0f9d755e9e49d255b09cb96a73ca80735b93a6d55ef1db19bb737ff6b WatchSource:0}: Error finding container f89e87a0f9d755e9e49d255b09cb96a73ca80735b93a6d55ef1db19bb737ff6b: Status 404 returned error can't find the container with id f89e87a0f9d755e9e49d255b09cb96a73ca80735b93a6d55ef1db19bb737ff6b
Oct 30 19:41:30.562661 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:30.562624    2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:41:30.916359 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:30.916182    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret\") pod \"global-pull-secret-syncer-8t48c\" (UID: \"93cfc69d-646b-4910-972d-04c9076c6c3c\") " pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:30.916359 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:30.916227    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs\") pod \"network-metrics-daemon-rj77q\" (UID: \"9cd48d14-cd97-4894-a43e-fa678a1833b3\") " pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:30.916359 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:30.916327    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:41:30.916359 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:30.916340    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:41:30.916685 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:30.916394    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret podName:93cfc69d-646b-4910-972d-04c9076c6c3c nodeName:}" failed. No retries permitted until 2026-10-30 19:41:32.916375326 +0000 UTC m=+5.261254806 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret") pod "global-pull-secret-syncer-8t48c" (UID: "93cfc69d-646b-4910-972d-04c9076c6c3c") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:41:30.916685 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:30.916414    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs podName:9cd48d14-cd97-4894-a43e-fa678a1833b3 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:32.916404974 +0000 UTC m=+5.261284442 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs") pod "network-metrics-daemon-rj77q" (UID: "9cd48d14-cd97-4894-a43e-fa678a1833b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:41:31.017281 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:31.017246    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2tsl\" (UniqueName: \"kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl\") pod \"network-check-target-m9t6w\" (UID: \"14e5d4c8-c53b-43b7-8301-804226ba1867\") " pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:41:31.017776 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:31.017426    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:41:31.017776 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:31.017465    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:41:31.017776 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:31.017478    2573 projected.go:196] Error preparing data for projected volume kube-api-access-v2tsl for pod openshift-network-diagnostics/network-check-target-m9t6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:41:31.017776 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:31.017531    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl podName:14e5d4c8-c53b-43b7-8301-804226ba1867 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:33.01751415 +0000 UTC m=+5.362393635 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2tsl" (UniqueName: "kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl") pod "network-check-target-m9t6w" (UID: "14e5d4c8-c53b-43b7-8301-804226ba1867") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:41:31.130350 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:31.130318    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:31.130554 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:31.130436    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:41:31.130554 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:31.130544    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:31.130674 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:31.130623    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:41:31.130785 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:31.130696    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:41:31.130785 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:31.130762    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:41:31.142308 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:31.142278    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dxw48" event={"ID":"f7a30ded-2282-462e-bd34-dd4ad735a3ca","Type":"ContainerStarted","Data":"96ea2175c027fdae0036df0774cf8de40dc933e8677e3cf7af9de89078144182"}
Oct 30 19:41:31.143886 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:31.143861    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qz2rj" event={"ID":"043ccaed-b00b-4406-b156-eb95ea8e05f3","Type":"ContainerStarted","Data":"c230cebc2ac527984ae7be71fd32fe26330fc871ccc702858134fb1848cf8146"}
Oct 30 19:41:31.145680 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:31.145656    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rwbsr" event={"ID":"ba9ed4cb-13c2-4ead-b549-5469e2724f6c","Type":"ContainerStarted","Data":"f89e87a0f9d755e9e49d255b09cb96a73ca80735b93a6d55ef1db19bb737ff6b"}
Oct 30 19:41:31.149588 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:31.149564    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snlg9" event={"ID":"21e03037-74dd-44b2-bac7-791d30cf240d","Type":"ContainerStarted","Data":"220752f888ce8a936d7ee2a4a0ec9c6b58882fb3286025b389130741ffc82a15"}
Oct 30 19:41:31.152513 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:31.152489    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg" event={"ID":"023e1dd6-ffb9-48c8-a407-be580b23bddb","Type":"ContainerStarted","Data":"b31aeda5cb15fecf77cf84227228897b19991c47a8ee13fdea3b6aa2bcde28a7"}
Oct 30 19:41:31.312655 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:31.312521    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:36:29 +0000 UTC" deadline="2028-03-28 23:05:51.539757015 +0000 UTC"
Oct 30 19:41:31.312655 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:31.312580    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12363h24m20.227181002s"
Oct 30 19:41:32.933472 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:32.933407    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs\") pod \"network-metrics-daemon-rj77q\" (UID: \"9cd48d14-cd97-4894-a43e-fa678a1833b3\") " pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:32.934122 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:32.933559    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret\") pod \"global-pull-secret-syncer-8t48c\" (UID: \"93cfc69d-646b-4910-972d-04c9076c6c3c\") " pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:32.934122 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:32.933681    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:41:32.934122 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:32.933772    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret podName:93cfc69d-646b-4910-972d-04c9076c6c3c nodeName:}" failed. No retries permitted until 2026-10-30 19:41:36.933722122 +0000 UTC m=+9.278601592 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret") pod "global-pull-secret-syncer-8t48c" (UID: "93cfc69d-646b-4910-972d-04c9076c6c3c") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:41:32.934278 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:32.934176    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:41:32.934278 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:32.934214    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs podName:9cd48d14-cd97-4894-a43e-fa678a1833b3 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:36.934203113 +0000 UTC m=+9.279082580 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs") pod "network-metrics-daemon-rj77q" (UID: "9cd48d14-cd97-4894-a43e-fa678a1833b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:41:33.034740 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:33.034696    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2tsl\" (UniqueName: \"kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl\") pod \"network-check-target-m9t6w\" (UID: \"14e5d4c8-c53b-43b7-8301-804226ba1867\") " pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:41:33.034909 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:33.034849    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:41:33.034909 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:33.034868    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:41:33.034909 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:33.034881    2573 projected.go:196] Error preparing data for projected volume kube-api-access-v2tsl for pod openshift-network-diagnostics/network-check-target-m9t6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:41:33.035059 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:33.034936    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl podName:14e5d4c8-c53b-43b7-8301-804226ba1867 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:37.034919069 +0000 UTC m=+9.379798542 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2tsl" (UniqueName: "kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl") pod "network-check-target-m9t6w" (UID: "14e5d4c8-c53b-43b7-8301-804226ba1867") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:41:33.129687 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:33.129653    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:33.129860 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:33.129776    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:41:33.129860 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:33.129842    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:33.130020 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:33.129916    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:41:33.130020 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:33.129966    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:41:33.130124 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:33.130032    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:41:35.130396 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:35.129519    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:41:35.130396 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:35.129855    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:41:35.130396 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:35.129915    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:35.130396 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:35.129976    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:41:35.130396 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:35.130005    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:35.130396 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:35.130079    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:41:35.177251 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:35.177217    2573 generic.go:356] "Generic (PLEG): container finished" podID="ce4e2caa0da71bc2cbc3d09da16fb366" containerID="83694aedc05138d37a3b8060b490230c635d02b7bfe840e27b996a719aa83bc5" exitCode=0
Oct 30 19:41:35.177366 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:35.177309    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal" event={"ID":"ce4e2caa0da71bc2cbc3d09da16fb366","Type":"ContainerDied","Data":"83694aedc05138d37a3b8060b490230c635d02b7bfe840e27b996a719aa83bc5"}
Oct 30 19:41:35.195650 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:35.194385    2573 generic.go:356] "Generic (PLEG): container finished" podID="21e03037-74dd-44b2-bac7-791d30cf240d" containerID="4e422d7ace23a35ebcfb19012524b52e0842c53ff40bfc8b57fced0bfea0c845" exitCode=0
Oct 30 19:41:35.195650 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:35.194502    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snlg9" event={"ID":"21e03037-74dd-44b2-bac7-791d30cf240d","Type":"ContainerDied","Data":"4e422d7ace23a35ebcfb19012524b52e0842c53ff40bfc8b57fced0bfea0c845"}
Oct 30 19:41:35.200085 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:35.199598    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal" event={"ID":"84c50bb941905481e1c090101140d6a9","Type":"ContainerStarted","Data":"779c78b1607fc2a35a721852f8b8545ca3c80556f73164d1c93a1271bed817c8"}
Oct 30 19:41:35.422438 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:35.422363    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal" podStartSLOduration=7.422343026 podStartE2EDuration="7.422343026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:35.421869487 +0000 UTC m=+7.766748975" watchObservedRunningTime="2026-10-30 19:41:35.422343026 +0000 UTC m=+7.767222522"
Oct 30 19:41:36.203720 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:36.203677    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9x594" event={"ID":"6391e2a6-bb94-4270-9fa9-f6d35f3e433c","Type":"ContainerStarted","Data":"eb36f0a03bd3b0e2347741fe5c1605dbd72576c7bee379ccf165c77387158f39"}
Oct 30 19:41:36.207142 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:36.207112    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal" event={"ID":"ce4e2caa0da71bc2cbc3d09da16fb366","Type":"ContainerStarted","Data":"b33c8243ee6e5bdff8f7299ba6c0d27065dc647181ddc5e63f1040702d34f069"}
Oct 30 19:41:36.209655 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:36.209629    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rwbsr" event={"ID":"ba9ed4cb-13c2-4ead-b549-5469e2724f6c","Type":"ContainerStarted","Data":"a26f9dfa5eccb06d4b1274f45b1012a088447c94e1a245e54a0b5434dfe4f71a"}
Oct 30 19:41:36.214257 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:36.214236    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52" event={"ID":"08a234af-5c01-4901-80ae-2ec70fa08bfe","Type":"ContainerStarted","Data":"610a11630647cec3819b9b26c432a493983ac096cc280f92f8b0d885893dd0d4"}
Oct 30 19:41:36.447851 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:36.447790    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9x594" podStartSLOduration=3.169442341 podStartE2EDuration="8.447769422s" totalImagesPullingTime="5.278327081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:36.325170298 +0000 UTC m=+8.670049786" watchObservedRunningTime="2026-10-30 19:41:36.447769422 +0000 UTC m=+8.792648910"
Oct 30 19:41:36.568850 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:36.568800    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rwbsr" podStartSLOduration=4.120615281 podStartE2EDuration="8.56878379s" totalImagesPullingTime="4.448168509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:36.448168751 +0000 UTC m=+8.793048262" watchObservedRunningTime="2026-10-30 19:41:36.56878379 +0000 UTC m=+8.913663276"
Oct 30 19:41:36.966958 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:36.966879    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret\") pod \"global-pull-secret-syncer-8t48c\" (UID: \"93cfc69d-646b-4910-972d-04c9076c6c3c\") " pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:36.966958 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:36.966935    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs\") pod \"network-metrics-daemon-rj77q\" (UID: \"9cd48d14-cd97-4894-a43e-fa678a1833b3\") " pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:36.967210 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:36.967060    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:41:36.967210 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:36.967120    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs podName:9cd48d14-cd97-4894-a43e-fa678a1833b3 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:44.96710142 +0000 UTC m=+17.311980890 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs") pod "network-metrics-daemon-rj77q" (UID: "9cd48d14-cd97-4894-a43e-fa678a1833b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:41:36.967559 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:36.967530    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:41:36.967722 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:36.967590    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret podName:93cfc69d-646b-4910-972d-04c9076c6c3c nodeName:}" failed. No retries permitted until 2026-10-30 19:41:44.967573419 +0000 UTC m=+17.312452904 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret") pod "global-pull-secret-syncer-8t48c" (UID: "93cfc69d-646b-4910-972d-04c9076c6c3c") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:41:37.068216 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:37.068169    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2tsl\" (UniqueName: \"kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl\") pod \"network-check-target-m9t6w\" (UID: \"14e5d4c8-c53b-43b7-8301-804226ba1867\") " pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:41:37.068384 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:37.068356    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:41:37.068384 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:37.068374    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:41:37.068501 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:37.068385    2573 projected.go:196] Error preparing data for projected volume kube-api-access-v2tsl for pod openshift-network-diagnostics/network-check-target-m9t6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:41:37.068501 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:37.068442    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl podName:14e5d4c8-c53b-43b7-8301-804226ba1867 nodeName:}" failed. No retries permitted until 2026-10-30 19:41:45.068423213 +0000 UTC m=+17.413302693 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2tsl" (UniqueName: "kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl") pod "network-check-target-m9t6w" (UID: "14e5d4c8-c53b-43b7-8301-804226ba1867") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:41:37.130587 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:37.130269    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:41:37.130587 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:37.130401    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:41:37.130587 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:37.130486    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:37.130587 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:37.130557    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:41:37.130917 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:37.130610    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:37.130917 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:37.130676    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:41:38.074859 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:38.074815    2573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9838bc_8604_40f5_99d5_70c240eb439d.slice/crio-ef93fc01d28476b86953c38978c0270f8fea8fede0170cbf3b2045116e2cc5f0.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:41:38.126943 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:38.126909    2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:41:38.221764 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:38.221695    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5qzfj" event={"ID":"9b9838bc-8604-40f5-99d5-70c240eb439d","Type":"ContainerStarted","Data":"ef93fc01d28476b86953c38978c0270f8fea8fede0170cbf3b2045116e2cc5f0"}
Oct 30 19:41:38.225108 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:38.224936    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l2mtk" event={"ID":"1c4d99d2-8ba6-410c-8337-f527e98d4651","Type":"ContainerStarted","Data":"2d8069fef71beac80aa2934f2321b2d3362e041628b2fed5a46279b1c1f08812"}
Oct 30 19:41:38.227211 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:38.227187    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52" event={"ID":"08a234af-5c01-4901-80ae-2ec70fa08bfe","Type":"ContainerStarted","Data":"8e99950e91ca3bbd9b79d1e1ce3e2443554343b7b25dc480e5bb696b54afdc9d"}
Oct 30 19:41:38.381175 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:38.381117    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal" podStartSLOduration=10.381099421 podStartE2EDuration="10.381099421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:36.571196889 +0000 UTC m=+8.916076379" watchObservedRunningTime="2026-10-30 19:41:38.381099421 +0000 UTC m=+10.725978910"
Oct 30 19:41:38.502820 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:38.502729    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-l2mtk" podStartSLOduration=2.585571425 podStartE2EDuration="10.502713094s" totalImagesPullingTime="7.917141669s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:38.50246505 +0000 UTC m=+10.847344535" watchObservedRunningTime="2026-10-30 19:41:38.502713094 +0000 UTC m=+10.847592585"
Oct 30 19:41:38.502968 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:38.502871    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-5qzfj" podStartSLOduration=2.349246953 podStartE2EDuration="10.502865522s" totalImagesPullingTime="8.153618569s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:38.382030823 +0000 UTC m=+10.726910313" watchObservedRunningTime="2026-10-30 19:41:38.502865522 +0000 UTC m=+10.847745009"
Oct 30 19:41:39.065258 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:39.065174    2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:41:38.126932129Z","UUID":"1bff0fab-1697-42bd-abfc-0a6230ff0339","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:41:39.066971 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:39.066681    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:41:39.066971 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:39.066714    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:41:39.130539 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:39.130511    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:39.130939 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:39.130652    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:41:39.131065 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:39.131032    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:39.131153 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:39.131135    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:41:39.131224 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:39.131200    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:41:39.131276 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:39.131252    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:41:39.234613 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:39.234547    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snlg9" event={"ID":"21e03037-74dd-44b2-bac7-791d30cf240d","Type":"ContainerStarted","Data":"20c327eb7f04b6b3110fb9250fbfeca9bf224d39e5001d8b0dc758db20881812"}
Oct 30 19:41:40.237843 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:40.237768    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dxw48" event={"ID":"f7a30ded-2282-462e-bd34-dd4ad735a3ca","Type":"ContainerStarted","Data":"1149baf8aa1942bfc8020e9c86b1301c0f3716713abbbe77fd2c8e127129811d"}
Oct 30 19:41:40.240157 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:40.240114    2573 generic.go:356] "Generic (PLEG): container finished" podID="21e03037-74dd-44b2-bac7-791d30cf240d" containerID="20c327eb7f04b6b3110fb9250fbfeca9bf224d39e5001d8b0dc758db20881812" exitCode=0
Oct 30 19:41:40.240295 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:40.240217    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snlg9" event={"ID":"21e03037-74dd-44b2-bac7-791d30cf240d","Type":"ContainerDied","Data":"20c327eb7f04b6b3110fb9250fbfeca9bf224d39e5001d8b0dc758db20881812"}
Oct 30 19:41:40.242674 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:40.242648    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52" event={"ID":"08a234af-5c01-4901-80ae-2ec70fa08bfe","Type":"ContainerStarted","Data":"bba77b9d32777139d634a093992f54c81c79b73e84599346f330f9d19381a4ef"}
Oct 30 19:41:40.497279 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:40.497227    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dxw48" podStartSLOduration=4.375223685 podStartE2EDuration="12.497212882s" totalImagesPullingTime="8.121989197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:40.356982341 +0000 UTC m=+12.701861829" watchObservedRunningTime="2026-10-30 19:41:40.497212882 +0000 UTC m=+12.842092369"
Oct 30 19:41:40.499284 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:40.499262    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rwbsr"
Oct 30 19:41:40.499790 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:40.499770    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rwbsr"
Oct 30 19:41:40.632763 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:40.632710    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52" podStartSLOduration=3.016295938 podStartE2EDuration="12.632694382s" totalImagesPullingTime="9.616398444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:40.632208582 +0000 UTC m=+12.977088071" watchObservedRunningTime="2026-10-30 19:41:40.632694382 +0000 UTC m=+12.977573869"
Oct 30 19:41:41.129804 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:41.129768    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:41:41.129982 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:41.129812    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:41.129982 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:41.129869    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:41.129982 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:41.129880    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:41:41.129982 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:41.129957    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:41:41.130229 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:41.130030    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:41:41.247294 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:41.247256    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qz2rj" event={"ID":"043ccaed-b00b-4406-b156-eb95ea8e05f3","Type":"ContainerStarted","Data":"619bd34b70291aa749cc29f69efe53de89594934a950dea85e958dd0b487d16f"}
Oct 30 19:41:41.879436 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:41.879402    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rwbsr"
Oct 30 19:41:41.880020 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:41.879993    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rwbsr"
Oct 30 19:41:42.002492 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:42.002427    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qz2rj" podStartSLOduration=4.055497675 podStartE2EDuration="14.002411351s" totalImagesPullingTime="9.946913676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:41.385089955 +0000 UTC m=+13.729969442" watchObservedRunningTime="2026-10-30 19:41:42.002411351 +0000 UTC m=+14.347290840"
Oct 30 19:41:42.251860 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:42.251781    2573 generic.go:356] "Generic (PLEG): container finished" podID="21e03037-74dd-44b2-bac7-791d30cf240d" containerID="17ebe86da4b369a2ef173f3eecebd573696dc1812fe83cc0c52a81b3258a8b27" exitCode=0
Oct 30 19:41:42.252411 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:42.251858    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snlg9" event={"ID":"21e03037-74dd-44b2-bac7-791d30cf240d","Type":"ContainerDied","Data":"17ebe86da4b369a2ef173f3eecebd573696dc1812fe83cc0c52a81b3258a8b27"}
Oct 30 19:41:43.129878 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:43.129833    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:43.129878 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:43.129878    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:41:43.130128 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:43.129968    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:43.130128 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:43.129975    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:41:43.130128 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:43.130081    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:41:43.130246 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:43.130150    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:41:44.259122 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:44.258916    2573 generic.go:356] "Generic (PLEG): container finished" podID="21e03037-74dd-44b2-bac7-791d30cf240d" containerID="7cce0678bfa61b6a91d3486d915c1ca32b7ad56b201c48a5c4770c50aabe582d" exitCode=0
Oct 30 19:41:44.259592 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:44.258995    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snlg9" event={"ID":"21e03037-74dd-44b2-bac7-791d30cf240d","Type":"ContainerDied","Data":"7cce0678bfa61b6a91d3486d915c1ca32b7ad56b201c48a5c4770c50aabe582d"}
Oct 30 19:41:45.030256 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:45.030231    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret\") pod \"global-pull-secret-syncer-8t48c\" (UID: \"93cfc69d-646b-4910-972d-04c9076c6c3c\") " pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:45.030370 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:45.030268    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs\") pod \"network-metrics-daemon-rj77q\" (UID: \"9cd48d14-cd97-4894-a43e-fa678a1833b3\") " pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:45.030402 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:45.030381    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:41:45.030441 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:45.030432    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret podName:93cfc69d-646b-4910-972d-04c9076c6c3c nodeName:}" failed. No retries permitted until 2026-10-30 19:42:01.030419558 +0000 UTC m=+33.375299037 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret") pod "global-pull-secret-syncer-8t48c" (UID: "93cfc69d-646b-4910-972d-04c9076c6c3c") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:41:45.030502 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:45.030381    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:41:45.030502 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:45.030494    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs podName:9cd48d14-cd97-4894-a43e-fa678a1833b3 nodeName:}" failed. No retries permitted until 2026-10-30 19:42:01.03048473 +0000 UTC m=+33.375364198 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs") pod "network-metrics-daemon-rj77q" (UID: "9cd48d14-cd97-4894-a43e-fa678a1833b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:41:45.130262 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:45.130239    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:41:45.130349 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:45.130328    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:45.130349 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:45.130335    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:41:45.130427 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:45.130345    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:45.130480 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:45.130433    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:41:45.133161 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:45.130838    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2tsl\" (UniqueName: \"kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl\") pod \"network-check-target-m9t6w\" (UID: \"14e5d4c8-c53b-43b7-8301-804226ba1867\") " pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:41:45.133161 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:45.130935    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:41:45.133161 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:45.131038    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:41:45.133161 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:45.131061    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:41:45.133161 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:45.131081    2573 projected.go:196] Error preparing data for projected volume kube-api-access-v2tsl for pod openshift-network-diagnostics/network-check-target-m9t6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:41:45.133161 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:45.131141    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl podName:14e5d4c8-c53b-43b7-8301-804226ba1867 nodeName:}" failed. No retries permitted until 2026-10-30 19:42:01.131118355 +0000 UTC m=+33.475997823 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2tsl" (UniqueName: "kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl") pod "network-check-target-m9t6w" (UID: "14e5d4c8-c53b-43b7-8301-804226ba1867") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:41:45.265142 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:45.264914    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 19:41:45.265440 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:45.265336    2573 generic.go:356] "Generic (PLEG): container finished" podID="023e1dd6-ffb9-48c8-a407-be580b23bddb" containerID="d4033ab8d6dea9638d45016197f050f6c2301cba6416845eaf1b8cefc629562c" exitCode=1
Oct 30 19:41:45.265440 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:45.265384    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg" event={"ID":"023e1dd6-ffb9-48c8-a407-be580b23bddb","Type":"ContainerStarted","Data":"af37d174a0e26072fe146655cb1f17949f4a560665447e9a35c86e171370babc"}
Oct 30 19:41:45.265562 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:45.265484    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg" event={"ID":"023e1dd6-ffb9-48c8-a407-be580b23bddb","Type":"ContainerStarted","Data":"e11609dbdbd351203f063317eca9c3ec8fbe8fc186e0c78b3eccbee2966c5894"}
Oct 30 19:41:45.265562 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:45.265538    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg" event={"ID":"023e1dd6-ffb9-48c8-a407-be580b23bddb","Type":"ContainerDied","Data":"d4033ab8d6dea9638d45016197f050f6c2301cba6416845eaf1b8cefc629562c"}
Oct 30 19:41:45.265631 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:45.265584    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg" event={"ID":"023e1dd6-ffb9-48c8-a407-be580b23bddb","Type":"ContainerStarted","Data":"64563822bea9d206cf14bbc9ce8bfeb9d74d1f1089d848f2d862c370d5cf6f52"}
Oct 30 19:41:46.270148 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:46.269980    2573 generic.go:356] "Generic (PLEG): container finished" podID="21e03037-74dd-44b2-bac7-791d30cf240d" containerID="b88fdc18114d418ea6aaaf0a9767d0dcc137368b14a6725d9213a635bd3681a8" exitCode=0
Oct 30 19:41:46.270529 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:46.270052    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snlg9" event={"ID":"21e03037-74dd-44b2-bac7-791d30cf240d","Type":"ContainerDied","Data":"b88fdc18114d418ea6aaaf0a9767d0dcc137368b14a6725d9213a635bd3681a8"}
Oct 30 19:41:46.272867 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:46.272851    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 19:41:46.273165 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:46.273147    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg" event={"ID":"023e1dd6-ffb9-48c8-a407-be580b23bddb","Type":"ContainerStarted","Data":"2d1c5b964563d7117317680bc056ad035ff837f58424ae9ae6cf364adfe217ac"}
Oct 30 19:41:46.273258 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:46.273204    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg" event={"ID":"023e1dd6-ffb9-48c8-a407-be580b23bddb","Type":"ContainerStarted","Data":"4133fd1d362b0ee1c97c4f4d856fe6c5d70e5a911357d6cb1847edf25c7b875b"}
Oct 30 19:41:46.538046 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:46.538020    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rwbsr_ba9ed4cb-13c2-4ead-b549-5469e2724f6c/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:47.130236 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:47.130207    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:47.130236 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:47.130238    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:47.130418 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:47.130309    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:41:47.130418 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:47.130370    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:41:47.130502 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:47.130438    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:41:47.130540 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:47.130504    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:41:47.277995 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:47.277963    2573 generic.go:356] "Generic (PLEG): container finished" podID="21e03037-74dd-44b2-bac7-791d30cf240d" containerID="2dff488a7e1fa75df4990bb69934e7de07ef114c525dac2113b18e4dab359061" exitCode=0
Oct 30 19:41:47.278397 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:47.278007    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snlg9" event={"ID":"21e03037-74dd-44b2-bac7-791d30cf240d","Type":"ContainerDied","Data":"2dff488a7e1fa75df4990bb69934e7de07ef114c525dac2113b18e4dab359061"}
Oct 30 19:41:48.283170 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:48.283127    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snlg9" event={"ID":"21e03037-74dd-44b2-bac7-791d30cf240d","Type":"ContainerStarted","Data":"0bf7b3f0fed94c116d70a995af4d7a7a5a6a5d4dabd8ae6073d5f203e1af861d"}
Oct 30 19:41:48.285851 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:48.285832    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 19:41:48.286175 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:48.286158    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg" event={"ID":"023e1dd6-ffb9-48c8-a407-be580b23bddb","Type":"ContainerStarted","Data":"e4e1431d4579214dd6a57228724f757542c59848266eeeb15fe60d56e4b22971"}
Oct 30 19:41:48.460994 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:48.460945    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-snlg9" podStartSLOduration=8.924082122 podStartE2EDuration="20.460932568s" totalImagesPullingTime="11.536850446s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:48.460927914 +0000 UTC m=+20.805807402" watchObservedRunningTime="2026-10-30 19:41:48.460932568 +0000 UTC m=+20.805812056"
Oct 30 19:41:49.129545 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:49.129513    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:41:49.129708 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:49.129513    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:49.129708 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:49.129621    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:49.129708 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:49.129690    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:41:49.129708 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:49.129594    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:41:49.129851 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:49.129768    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:41:50.294121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:50.293968    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 19:41:50.294765 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:50.294461    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg" event={"ID":"023e1dd6-ffb9-48c8-a407-be580b23bddb","Type":"ContainerStarted","Data":"85c92ea5e01a931338a62ccfa0f9f6e92cf8bedf1a161e9cd4db8ec5e5f570b1"}
Oct 30 19:41:50.294765 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:50.294750    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:50.294896 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:50.294807    2573 scope.go:122] "RemoveContainer" containerID="d4033ab8d6dea9638d45016197f050f6c2301cba6416845eaf1b8cefc629562c"
Oct 30 19:41:50.312126 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:50.312105    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:51.129971 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:51.129941    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:41:51.130225 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:51.129993    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:51.130225 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:51.130077    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:41:51.130225 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:51.130088    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:51.130225 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:51.130174    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:41:51.130348 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:51.130229    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:41:51.300160 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:51.300137    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 19:41:51.300588 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:51.300485    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg" event={"ID":"023e1dd6-ffb9-48c8-a407-be580b23bddb","Type":"ContainerStarted","Data":"0219f62faed8044fab1302a20598c49dd334c042bc10f473a655346241a8d957"}
Oct 30 19:41:51.300588 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:51.300568    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:41:51.300726 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:51.300710    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:51.315404 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:51.315386    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:51.435019 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:51.434931    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg" podStartSLOduration=9.045350588 podStartE2EDuration="23.434918081s" totalImagesPullingTime="14.389567493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:51.434860407 +0000 UTC m=+23.779739895" watchObservedRunningTime="2026-10-30 19:41:51.434918081 +0000 UTC m=+23.779797569"
Oct 30 19:41:52.302571 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:52.302544    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:41:52.703827 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:52.703800    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5qzfj_9b9838bc-8604-40f5-99d5-70c240eb439d/dns-node-resolver/0.log"
Oct 30 19:41:53.129873 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:53.129842    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:41:53.129873 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:53.129859    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:53.130115 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:53.129963    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:53.130115 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:53.129968    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:41:53.130115 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:53.130041    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:41:53.130236 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:53.130111    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:41:53.304850 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:53.304826    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:41:53.502656 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:53.502589    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9x594_6391e2a6-bb94-4270-9fa9-f6d35f3e433c/node-ca/0.log"
Oct 30 19:41:55.130183 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:55.130155    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:41:55.130627 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:55.130155    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:55.130627 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:55.130153    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:55.130627 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:55.130350    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:41:55.130627 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:55.130250    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:41:55.130627 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:55.130410    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:41:57.129595 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:57.129569    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:41:57.129969 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:57.129635    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:57.129969 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:57.129721    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:41:57.129969 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:57.129727    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:57.129969 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:57.129796    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:41:57.129969 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:57.129857    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:41:58.375479 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:58.375435    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:41:58.375955 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:58.375647    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:41:58.390021 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:58.389989    2573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg" podUID="023e1dd6-ffb9-48c8-a407-be580b23bddb" containerName="ovnkube-controller" probeResult="failure" output=""
Oct 30 19:41:58.398380 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:58.398356    2573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg" podUID="023e1dd6-ffb9-48c8-a407-be580b23bddb" containerName="ovnkube-controller" probeResult="failure" output=""
Oct 30 19:41:59.129897 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:59.129865    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:41:59.129897 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:59.129878    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:41:59.129897 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:41:59.129893    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:41:59.130151 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:59.129957    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:41:59.130151 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:59.130023    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:41:59.130151 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:41:59.130090    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:42:01.053416 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:01.053384    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret\") pod \"global-pull-secret-syncer-8t48c\" (UID: \"93cfc69d-646b-4910-972d-04c9076c6c3c\") " pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:42:01.053828 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:01.053422    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs\") pod \"network-metrics-daemon-rj77q\" (UID: \"9cd48d14-cd97-4894-a43e-fa678a1833b3\") " pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:42:01.053828 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:01.053558    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:42:01.053828 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:01.053558    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:42:01.053828 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:01.053617    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs podName:9cd48d14-cd97-4894-a43e-fa678a1833b3 nodeName:}" failed. No retries permitted until 2026-10-30 19:42:33.053602609 +0000 UTC m=+65.398482078 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs") pod "network-metrics-daemon-rj77q" (UID: "9cd48d14-cd97-4894-a43e-fa678a1833b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:42:01.053828 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:01.053631    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret podName:93cfc69d-646b-4910-972d-04c9076c6c3c nodeName:}" failed. No retries permitted until 2026-10-30 19:42:33.053625401 +0000 UTC m=+65.398504867 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret") pod "global-pull-secret-syncer-8t48c" (UID: "93cfc69d-646b-4910-972d-04c9076c6c3c") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:42:01.129874 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:01.129845    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:42:01.130021 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:01.129874    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:42:01.130021 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:01.129956    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:42:01.130021 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:01.129980    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:42:01.130166 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:01.130050    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:42:01.130166 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:01.130124    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:42:01.154048 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:01.154025    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2tsl\" (UniqueName: \"kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl\") pod \"network-check-target-m9t6w\" (UID: \"14e5d4c8-c53b-43b7-8301-804226ba1867\") " pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:42:01.154139 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:01.154127    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:42:01.154176 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:01.154145    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:42:01.154176 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:01.154154    2573 projected.go:196] Error preparing data for projected volume kube-api-access-v2tsl for pod openshift-network-diagnostics/network-check-target-m9t6w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:42:01.154243 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:01.154193    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl podName:14e5d4c8-c53b-43b7-8301-804226ba1867 nodeName:}" failed. No retries permitted until 2026-10-30 19:42:33.154180974 +0000 UTC m=+65.499060441 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2tsl" (UniqueName: "kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl") pod "network-check-target-m9t6w" (UID: "14e5d4c8-c53b-43b7-8301-804226ba1867") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:42:03.130299 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:03.130263    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:42:03.130299 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:03.130290    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:42:03.130730 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:03.130380    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:42:03.130730 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:03.130422    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:42:03.130730 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:03.130518    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:42:03.130730 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:03.130548    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:42:05.129570 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:05.129538    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:42:05.129956 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:05.129582    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:42:05.129956 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:05.129584    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:42:05.129956 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:05.129647    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:42:05.129956 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:05.129734    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:42:05.129956 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:05.129801    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:42:07.130336 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:07.130307    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:42:07.130336 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:07.130334    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:42:07.130821 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:07.130400    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:42:07.130821 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:07.130433    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:42:07.130821 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:07.130519    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:42:07.130821 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:07.130577    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:42:09.129848 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:09.129817    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:42:09.129848 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:09.129844    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:42:09.130284 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:09.129910    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:42:09.130284 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:09.129953    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:42:09.130284 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:09.130045    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:42:09.130284 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:09.130084    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:42:11.129641 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:11.129603    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:42:11.130024 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:11.129698    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:42:11.130024 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:11.129801    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:42:11.130024 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:11.129704    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:42:11.130024 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:11.129706    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:42:11.130024 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:11.129883    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:42:13.130326 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:13.130044    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:42:13.130778 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:13.130107    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:42:13.130778 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:13.130410    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:42:13.130778 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:13.130113    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:42:13.130778 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:13.130500    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:42:13.130778 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:13.130577    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:42:13.232012 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:13.231978    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8t48c"]
Oct 30 19:42:13.234927 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:13.234899    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m9t6w"]
Oct 30 19:42:13.235717 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:13.235692    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rj77q"]
Oct 30 19:42:13.345606 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:13.345581    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:42:13.345792 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:13.345581    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:42:13.345792 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:13.345673    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:42:13.345792 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:13.345769    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:42:13.345906 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:13.345581    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:42:13.345906 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:13.345874    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:42:15.129359 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:15.129329    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:42:15.129359 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:15.129347    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:42:15.129961 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:15.129331    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:42:15.129961 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:15.129440    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:42:15.129961 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:15.129679    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:42:15.129961 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:15.129708    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:42:17.130305 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:17.130265    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:42:17.130716 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:17.130280    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:42:17.130716 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:17.130392    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m9t6w" podUID="14e5d4c8-c53b-43b7-8301-804226ba1867"
Oct 30 19:42:17.130716 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:17.130401    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:42:17.130716 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:17.130514    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8t48c" podUID="93cfc69d-646b-4910-972d-04c9076c6c3c"
Oct 30 19:42:17.130716 ip-10-0-142-240 kubenswrapper[2573]: E1030 19:42:17.130584    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rj77q" podUID="9cd48d14-cd97-4894-a43e-fa678a1833b3"
Oct 30 19:42:18.369062 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.369042    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeReady"
Oct 30 19:42:18.369335 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.369168    2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 19:42:18.483648 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.483615    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b7jpb"]
Oct 30 19:42:18.485309 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.485288    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b7jpb"
Oct 30 19:42:18.486884 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.486861    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-t4b2n"]
Oct 30 19:42:18.488338 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.488319    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:18.539109 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.539079    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:42:18.539294 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.539179    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pxncz\""
Oct 30 19:42:18.540708 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.540692    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:42:18.540802 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.540789    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:42:18.543231 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.543215    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:42:18.543441 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.543426    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:42:18.543528 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.543439    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:42:18.545518 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.545504    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:42:18.547633 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.547620    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2wfbg\""
Oct 30 19:42:18.554851 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.554828    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t4b2n"]
Oct 30 19:42:18.557264 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.557244    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b7jpb"]
Oct 30 19:42:18.580911 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.580881    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-crio-socket\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:18.581057 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.580927    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:18.581057 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.581012    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-data-volume\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:18.581131 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.581069    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5afcafb9-baea-4a5f-906a-1ac08f1f6834-cert\") pod \"ingress-canary-b7jpb\" (UID: \"5afcafb9-baea-4a5f-906a-1ac08f1f6834\") " pod="openshift-ingress-canary/ingress-canary-b7jpb"
Oct 30 19:42:18.581131 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.581085    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5mbb\" (UniqueName: \"kubernetes.io/projected/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-kube-api-access-c5mbb\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:18.581131 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.581109    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:18.581131 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.581129    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzjng\" (UniqueName: \"kubernetes.io/projected/5afcafb9-baea-4a5f-906a-1ac08f1f6834-kube-api-access-qzjng\") pod \"ingress-canary-b7jpb\" (UID: \"5afcafb9-baea-4a5f-906a-1ac08f1f6834\") " pod="openshift-ingress-canary/ingress-canary-b7jpb"
Oct 30 19:42:18.594227 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.594198    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rjdlc"]
Oct 30 19:42:18.595834 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.595814    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rjdlc"
Oct 30 19:42:18.649035 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.649009    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:42:18.650311 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.650294    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qgpv7\""
Oct 30 19:42:18.650581 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.650564    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:42:18.658113 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.658081    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rjdlc"]
Oct 30 19:42:18.682247 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.682222    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5afcafb9-baea-4a5f-906a-1ac08f1f6834-cert\") pod \"ingress-canary-b7jpb\" (UID: \"5afcafb9-baea-4a5f-906a-1ac08f1f6834\") " pod="openshift-ingress-canary/ingress-canary-b7jpb"
Oct 30 19:42:18.682369 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.682251    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5mbb\" (UniqueName: \"kubernetes.io/projected/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-kube-api-access-c5mbb\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:18.682369 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.682271    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f31378b-b459-4ff0-9e72-d0d4266ab7b6-config-volume\") pod \"dns-default-rjdlc\" (UID: \"0f31378b-b459-4ff0-9e72-d0d4266ab7b6\") " pod="openshift-dns/dns-default-rjdlc"
Oct 30 19:42:18.682369 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.682303    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:18.682369 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.682325    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzjng\" (UniqueName: \"kubernetes.io/projected/5afcafb9-baea-4a5f-906a-1ac08f1f6834-kube-api-access-qzjng\") pod \"ingress-canary-b7jpb\" (UID: \"5afcafb9-baea-4a5f-906a-1ac08f1f6834\") " pod="openshift-ingress-canary/ingress-canary-b7jpb"
Oct 30 19:42:18.682537 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.682400    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-crio-socket\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:18.682587 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.682568    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f31378b-b459-4ff0-9e72-d0d4266ab7b6-metrics-tls\") pod \"dns-default-rjdlc\" (UID: \"0f31378b-b459-4ff0-9e72-d0d4266ab7b6\") " pod="openshift-dns/dns-default-rjdlc"
Oct 30 19:42:18.682638 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.682620    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-crio-socket\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:18.682705 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.682681    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f31378b-b459-4ff0-9e72-d0d4266ab7b6-tmp-dir\") pod \"dns-default-rjdlc\" (UID: \"0f31378b-b459-4ff0-9e72-d0d4266ab7b6\") " pod="openshift-dns/dns-default-rjdlc"
Oct 30 19:42:18.682753 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.682731    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:18.682811 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.682783    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-data-volume\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:18.682855 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.682813    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hklwg\" (UniqueName: \"kubernetes.io/projected/0f31378b-b459-4ff0-9e72-d0d4266ab7b6-kube-api-access-hklwg\") pod \"dns-default-rjdlc\" (UID: \"0f31378b-b459-4ff0-9e72-d0d4266ab7b6\") " pod="openshift-dns/dns-default-rjdlc"
Oct 30 19:42:18.683027 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.682993    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:18.683144 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.683046    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-data-volume\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:18.686333 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.686308    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:18.686410 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.686355    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5afcafb9-baea-4a5f-906a-1ac08f1f6834-cert\") pod \"ingress-canary-b7jpb\" (UID: \"5afcafb9-baea-4a5f-906a-1ac08f1f6834\") " pod="openshift-ingress-canary/ingress-canary-b7jpb"
Oct 30 19:42:18.746671 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.746642    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5mbb\" (UniqueName: \"kubernetes.io/projected/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-kube-api-access-c5mbb\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:18.746851 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.746832    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzjng\" (UniqueName: \"kubernetes.io/projected/5afcafb9-baea-4a5f-906a-1ac08f1f6834-kube-api-access-qzjng\") pod \"ingress-canary-b7jpb\" (UID: \"5afcafb9-baea-4a5f-906a-1ac08f1f6834\") " pod="openshift-ingress-canary/ingress-canary-b7jpb"
Oct 30 19:42:18.783463 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.783411    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f31378b-b459-4ff0-9e72-d0d4266ab7b6-metrics-tls\") pod \"dns-default-rjdlc\" (UID: \"0f31378b-b459-4ff0-9e72-d0d4266ab7b6\") " pod="openshift-dns/dns-default-rjdlc"
Oct 30 19:42:18.783463 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.783445    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f31378b-b459-4ff0-9e72-d0d4266ab7b6-tmp-dir\") pod \"dns-default-rjdlc\" (UID: \"0f31378b-b459-4ff0-9e72-d0d4266ab7b6\") " pod="openshift-dns/dns-default-rjdlc"
Oct 30 19:42:18.783649 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.783491    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hklwg\" (UniqueName: \"kubernetes.io/projected/0f31378b-b459-4ff0-9e72-d0d4266ab7b6-kube-api-access-hklwg\") pod \"dns-default-rjdlc\" (UID: \"0f31378b-b459-4ff0-9e72-d0d4266ab7b6\") " pod="openshift-dns/dns-default-rjdlc"
Oct 30 19:42:18.783649 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.783516    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f31378b-b459-4ff0-9e72-d0d4266ab7b6-config-volume\") pod \"dns-default-rjdlc\" (UID: \"0f31378b-b459-4ff0-9e72-d0d4266ab7b6\") " pod="openshift-dns/dns-default-rjdlc"
Oct 30 19:42:18.783856 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.783832    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f31378b-b459-4ff0-9e72-d0d4266ab7b6-tmp-dir\") pod \"dns-default-rjdlc\" (UID: \"0f31378b-b459-4ff0-9e72-d0d4266ab7b6\") " pod="openshift-dns/dns-default-rjdlc"
Oct 30 19:42:18.784060 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.784045    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f31378b-b459-4ff0-9e72-d0d4266ab7b6-config-volume\") pod \"dns-default-rjdlc\" (UID: \"0f31378b-b459-4ff0-9e72-d0d4266ab7b6\") " pod="openshift-dns/dns-default-rjdlc"
Oct 30 19:42:18.785804 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.785786    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f31378b-b459-4ff0-9e72-d0d4266ab7b6-metrics-tls\") pod \"dns-default-rjdlc\" (UID: \"0f31378b-b459-4ff0-9e72-d0d4266ab7b6\") " pod="openshift-dns/dns-default-rjdlc"
Oct 30 19:42:18.797091 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.797067    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b7jpb"
Oct 30 19:42:18.803372 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.803344    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:18.845743 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.845712    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hklwg\" (UniqueName: \"kubernetes.io/projected/0f31378b-b459-4ff0-9e72-d0d4266ab7b6-kube-api-access-hklwg\") pod \"dns-default-rjdlc\" (UID: \"0f31378b-b459-4ff0-9e72-d0d4266ab7b6\") " pod="openshift-dns/dns-default-rjdlc"
Oct 30 19:42:18.905753 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:18.905721    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rjdlc"
Oct 30 19:42:19.130193 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:19.130114    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:42:19.130193 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:19.130147    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:42:19.130193 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:19.130151    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:42:19.181793 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:19.181770    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b7jpb"]
Oct 30 19:42:19.184740 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:19.184712    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t4b2n"]
Oct 30 19:42:19.184993 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:42:19.184965    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5afcafb9_baea_4a5f_906a_1ac08f1f6834.slice/crio-addca54b0cfe92a90da331ab9549795785599e21e8aa7599b5e8c4bccaf399e5 WatchSource:0}: Error finding container addca54b0cfe92a90da331ab9549795785599e21e8aa7599b5e8c4bccaf399e5: Status 404 returned error can't find the container with id addca54b0cfe92a90da331ab9549795785599e21e8aa7599b5e8c4bccaf399e5
Oct 30 19:42:19.188734 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:42:19.188698    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b81aa0b_b68a_4225_81e6_0e5d59d5c63b.slice/crio-0ac9cbb47244a9352aa7d4bd1137c93bdbcc8c4991feb2789c0a17a5aef05c8f WatchSource:0}: Error finding container 0ac9cbb47244a9352aa7d4bd1137c93bdbcc8c4991feb2789c0a17a5aef05c8f: Status 404 returned error can't find the container with id 0ac9cbb47244a9352aa7d4bd1137c93bdbcc8c4991feb2789c0a17a5aef05c8f
Oct 30 19:42:19.190073 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:19.190052    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j4jh7\""
Oct 30 19:42:19.190159 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:19.190060    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2j2tt\""
Oct 30 19:42:19.190159 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:19.190114    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:42:19.190159 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:19.190134    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-vlbmc\""
Oct 30 19:42:19.190159 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:19.190148    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:42:19.190316 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:19.190154    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:42:19.190383 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:19.190372    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:42:19.279180 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:19.279153    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rjdlc"]
Oct 30 19:42:19.281367 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:42:19.281337    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f31378b_b459_4ff0_9e72_d0d4266ab7b6.slice/crio-3293696b27070ca5d0335ad20881602a8e9e78295151425f79e516c3d961d962 WatchSource:0}: Error finding container 3293696b27070ca5d0335ad20881602a8e9e78295151425f79e516c3d961d962: Status 404 returned error can't find the container with id 3293696b27070ca5d0335ad20881602a8e9e78295151425f79e516c3d961d962
Oct 30 19:42:19.358678 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:19.358644    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b7jpb" event={"ID":"5afcafb9-baea-4a5f-906a-1ac08f1f6834","Type":"ContainerStarted","Data":"addca54b0cfe92a90da331ab9549795785599e21e8aa7599b5e8c4bccaf399e5"}
Oct 30 19:42:19.359663 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:19.359636    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rjdlc" event={"ID":"0f31378b-b459-4ff0-9e72-d0d4266ab7b6","Type":"ContainerStarted","Data":"3293696b27070ca5d0335ad20881602a8e9e78295151425f79e516c3d961d962"}
Oct 30 19:42:19.361204 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:19.361182    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t4b2n" event={"ID":"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b","Type":"ContainerStarted","Data":"2d346a2c8f433ea2fdd0b91885400caca5dd761236f1e280b758b5435abe93ce"}
Oct 30 19:42:19.361288 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:19.361220    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t4b2n" event={"ID":"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b","Type":"ContainerStarted","Data":"0ac9cbb47244a9352aa7d4bd1137c93bdbcc8c4991feb2789c0a17a5aef05c8f"}
Oct 30 19:42:21.370368 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:21.370325    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t4b2n" event={"ID":"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b","Type":"ContainerStarted","Data":"dcc71537aa0a25953ea51195f8c97bf6221f0e52119385f59cf9accc87625ded"}
Oct 30 19:42:22.376114 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:22.375886    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b7jpb" event={"ID":"5afcafb9-baea-4a5f-906a-1ac08f1f6834","Type":"ContainerStarted","Data":"3f11b3a445ad93e6fb08357c56c4f1e3890de1e59306dc17c3ef031c8c0e2e40"}
Oct 30 19:42:22.378021 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:22.377980    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rjdlc" event={"ID":"0f31378b-b459-4ff0-9e72-d0d4266ab7b6","Type":"ContainerStarted","Data":"9e06bcb52d60f0053ea536a062f84bb1aa35ea96dbc188b58436fdc4c14964f0"}
Oct 30 19:42:22.378149 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:22.378035    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rjdlc" event={"ID":"0f31378b-b459-4ff0-9e72-d0d4266ab7b6","Type":"ContainerStarted","Data":"e7dcea0ca20201da009b165e8e2e558ded1aca97ede399ebca77dfc098c34457"}
Oct 30 19:42:22.378149 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:22.378134    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rjdlc"
Oct 30 19:42:22.501214 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:22.501171    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b7jpb" podStartSLOduration=1.437482319 podStartE2EDuration="4.50115772s" totalImagesPullingTime="3.063675401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:22.500090081 +0000 UTC m=+54.844969564" watchObservedRunningTime="2026-10-30 19:42:22.50115772 +0000 UTC m=+54.846037274"
Oct 30 19:42:23.029028 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.028972    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-rjdlc" podStartSLOduration=2.791289061 podStartE2EDuration="5.028956932s" totalImagesPullingTime="2.237667871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:22.620117475 +0000 UTC m=+54.964996962" watchObservedRunningTime="2026-10-30 19:42:23.028956932 +0000 UTC m=+55.373836470"
Oct 30 19:42:23.029399 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.029379    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl"]
Oct 30 19:42:23.031094 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.031074    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl"
Oct 30 19:42:23.093982 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.093904    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 19:42:23.093982 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.093943    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 19:42:23.137510 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.137473    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 19:42:23.137668 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.137518    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 19:42:23.137668 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.137557    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-7t4pl\""
Oct 30 19:42:23.137750 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.137736    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 19:42:23.144803 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.144773    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl"]
Oct 30 19:42:23.145366 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.145344    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"]
Oct 30 19:42:23.147326 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.147307    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:23.190256 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.190227    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-t586q"]
Oct 30 19:42:23.191979 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.191957    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"]
Oct 30 19:42:23.192107 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.192053    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.205308 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.205290    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 19:42:23.206021 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.206005    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-x99sr\""
Oct 30 19:42:23.206324 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.206303    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 19:42:23.206416 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.206323    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 19:42:23.218690 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.218673    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4drl\" (UniqueName: \"kubernetes.io/projected/a94cd52c-5a75-45c0-9669-587b56e8fc70-kube-api-access-k4drl\") pod \"openshift-state-metrics-7bd7489957-wd2gl\" (UID: \"a94cd52c-5a75-45c0-9669-587b56e8fc70\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl"
Oct 30 19:42:23.218764 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.218720    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a94cd52c-5a75-45c0-9669-587b56e8fc70-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wd2gl\" (UID: \"a94cd52c-5a75-45c0-9669-587b56e8fc70\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl"
Oct 30 19:42:23.218764 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.218758    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a94cd52c-5a75-45c0-9669-587b56e8fc70-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wd2gl\" (UID: \"a94cd52c-5a75-45c0-9669-587b56e8fc70\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl"
Oct 30 19:42:23.218900 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.218883    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a94cd52c-5a75-45c0-9669-587b56e8fc70-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wd2gl\" (UID: \"a94cd52c-5a75-45c0-9669-587b56e8fc70\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl"
Oct 30 19:42:23.268639 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.268612    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 19:42:23.305284 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.305261    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 19:42:23.305555 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.305540    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fn2xq\""
Oct 30 19:42:23.306031 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.306015    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 19:42:23.319569 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.319541    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a94cd52c-5a75-45c0-9669-587b56e8fc70-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wd2gl\" (UID: \"a94cd52c-5a75-45c0-9669-587b56e8fc70\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl"
Oct 30 19:42:23.319669 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.319585    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/419b608e-bfa8-4d2b-bacd-08c286fb7707-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:23.319669 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.319612    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j7rs\" (UniqueName: \"kubernetes.io/projected/419b608e-bfa8-4d2b-bacd-08c286fb7707-kube-api-access-4j7rs\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:23.319786 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.319764    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a94cd52c-5a75-45c0-9669-587b56e8fc70-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wd2gl\" (UID: \"a94cd52c-5a75-45c0-9669-587b56e8fc70\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl"
Oct 30 19:42:23.319816 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.319799    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-textfile\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.319844 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.319836    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhn8m\" (UniqueName: \"kubernetes.io/projected/56bbeb6c-dce1-42dc-9291-615c39153a7d-kube-api-access-hhn8m\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.319887 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.319861    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/56bbeb6c-dce1-42dc-9291-615c39153a7d-root\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.319932 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.319888    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.319975 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.319961    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-wtmp\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.320024 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.319996    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56bbeb6c-dce1-42dc-9291-615c39153a7d-sys\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.320079 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.320063    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/419b608e-bfa8-4d2b-bacd-08c286fb7707-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:23.320127 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.320106    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/419b608e-bfa8-4d2b-bacd-08c286fb7707-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:23.320171 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.320153    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-accelerators-collector-config\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.320225 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.320212    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a94cd52c-5a75-45c0-9669-587b56e8fc70-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wd2gl\" (UID: \"a94cd52c-5a75-45c0-9669-587b56e8fc70\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl"
Oct 30 19:42:23.320491 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.320469    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/419b608e-bfa8-4d2b-bacd-08c286fb7707-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:23.320609 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.320504    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4drl\" (UniqueName: \"kubernetes.io/projected/a94cd52c-5a75-45c0-9669-587b56e8fc70-kube-api-access-k4drl\") pod \"openshift-state-metrics-7bd7489957-wd2gl\" (UID: \"a94cd52c-5a75-45c0-9669-587b56e8fc70\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl"
Oct 30 19:42:23.320609 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.320529    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-tls\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.320718 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.320621    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/419b608e-bfa8-4d2b-bacd-08c286fb7707-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:23.320718 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.320681    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/56bbeb6c-dce1-42dc-9291-615c39153a7d-metrics-client-ca\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.320930 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.320909    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a94cd52c-5a75-45c0-9669-587b56e8fc70-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wd2gl\" (UID: \"a94cd52c-5a75-45c0-9669-587b56e8fc70\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl"
Oct 30 19:42:23.322158 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.322135    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a94cd52c-5a75-45c0-9669-587b56e8fc70-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wd2gl\" (UID: \"a94cd52c-5a75-45c0-9669-587b56e8fc70\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl"
Oct 30 19:42:23.322440 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.322420    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a94cd52c-5a75-45c0-9669-587b56e8fc70-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wd2gl\" (UID: \"a94cd52c-5a75-45c0-9669-587b56e8fc70\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl"
Oct 30 19:42:23.384487 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.384392    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4drl\" (UniqueName: \"kubernetes.io/projected/a94cd52c-5a75-45c0-9669-587b56e8fc70-kube-api-access-k4drl\") pod \"openshift-state-metrics-7bd7489957-wd2gl\" (UID: \"a94cd52c-5a75-45c0-9669-587b56e8fc70\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl"
Oct 30 19:42:23.386618 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.386591    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t4b2n" event={"ID":"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b","Type":"ContainerStarted","Data":"997ea183d2154e5db1fee62660ac44aba5630aa7750809e4f3ae913d121f13fb"}
Oct 30 19:42:23.421347 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.421307    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/56bbeb6c-dce1-42dc-9291-615c39153a7d-metrics-client-ca\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.421555 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.421366    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/419b608e-bfa8-4d2b-bacd-08c286fb7707-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:23.421555 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.421389    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j7rs\" (UniqueName: \"kubernetes.io/projected/419b608e-bfa8-4d2b-bacd-08c286fb7707-kube-api-access-4j7rs\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:23.421555 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.421426    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-textfile\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.421555 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.421477    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhn8m\" (UniqueName: \"kubernetes.io/projected/56bbeb6c-dce1-42dc-9291-615c39153a7d-kube-api-access-hhn8m\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.421555 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.421499    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/56bbeb6c-dce1-42dc-9291-615c39153a7d-root\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.421555 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.421521    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.421802 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.421618    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/56bbeb6c-dce1-42dc-9291-615c39153a7d-root\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.421802 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.421765    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-wtmp\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.421802 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.421789    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-textfile\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.421930 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.421806    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56bbeb6c-dce1-42dc-9291-615c39153a7d-sys\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.421930 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.421815    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/419b608e-bfa8-4d2b-bacd-08c286fb7707-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:23.421930 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.421846    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56bbeb6c-dce1-42dc-9291-615c39153a7d-sys\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.421930 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.421858    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/419b608e-bfa8-4d2b-bacd-08c286fb7707-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:23.421930 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.421905    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/419b608e-bfa8-4d2b-bacd-08c286fb7707-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:23.422147 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.421932    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-wtmp\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.422147 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.421949    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-accelerators-collector-config\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.422147 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.421978    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/419b608e-bfa8-4d2b-bacd-08c286fb7707-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:23.422147 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.421997    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/56bbeb6c-dce1-42dc-9291-615c39153a7d-metrics-client-ca\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.422147 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.422007    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-tls\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.422147 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.422043    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/419b608e-bfa8-4d2b-bacd-08c286fb7707-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:23.422494 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.422444    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-accelerators-collector-config\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.422618 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.422591    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/419b608e-bfa8-4d2b-bacd-08c286fb7707-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:23.422732 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.422627    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/419b608e-bfa8-4d2b-bacd-08c286fb7707-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:23.424231 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.424213    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-tls\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.424367 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.424346    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.424780 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.424756    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/419b608e-bfa8-4d2b-bacd-08c286fb7707-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:23.424848 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.424786    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/419b608e-bfa8-4d2b-bacd-08c286fb7707-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:23.487799 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.487767    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j7rs\" (UniqueName: \"kubernetes.io/projected/419b608e-bfa8-4d2b-bacd-08c286fb7707-kube-api-access-4j7rs\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:23.498846 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.498815    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhn8m\" (UniqueName: \"kubernetes.io/projected/56bbeb6c-dce1-42dc-9291-615c39153a7d-kube-api-access-hhn8m\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.504768 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.504748    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:23.513845 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:42:23.513817    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56bbeb6c_dce1_42dc_9291_615c39153a7d.slice/crio-f7a0c42e78195e1214da15d460376329d87d668b2a0e8568d6fd317c439b54c8 WatchSource:0}: Error finding container f7a0c42e78195e1214da15d460376329d87d668b2a0e8568d6fd317c439b54c8: Status 404 returned error can't find the container with id f7a0c42e78195e1214da15d460376329d87d668b2a0e8568d6fd317c439b54c8
Oct 30 19:42:23.577374 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.577321    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-t4b2n" podStartSLOduration=2.41256822 podStartE2EDuration="5.577306395s" totalImagesPullingTime="3.164738175s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:42:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:23.576916441 +0000 UTC m=+55.921795922" watchObservedRunningTime="2026-10-30 19:42:23.577306395 +0000 UTC m=+55.922185882"
Oct 30 19:42:23.642372 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.642294    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl"
Oct 30 19:42:23.759038 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:23.758982    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:24.084944 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:24.084911    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl"]
Oct 30 19:42:24.101911 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:42:24.101876    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda94cd52c_5a75_45c0_9669_587b56e8fc70.slice/crio-21b5386a6dae83c293ec44372355511c77f4f6b45f376989b85773bcbc034f3c WatchSource:0}: Error finding container 21b5386a6dae83c293ec44372355511c77f4f6b45f376989b85773bcbc034f3c: Status 404 returned error can't find the container with id 21b5386a6dae83c293ec44372355511c77f4f6b45f376989b85773bcbc034f3c
Oct 30 19:42:24.138725 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:24.138692    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"]
Oct 30 19:42:24.139043 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:42:24.139017    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod419b608e_bfa8_4d2b_bacd_08c286fb7707.slice/crio-8698545d4a79dded51e07962a78d7124c90fe8143c26dc272d56e3feb94ed846 WatchSource:0}: Error finding container 8698545d4a79dded51e07962a78d7124c90fe8143c26dc272d56e3feb94ed846: Status 404 returned error can't find the container with id 8698545d4a79dded51e07962a78d7124c90fe8143c26dc272d56e3feb94ed846
Oct 30 19:42:24.390852 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:24.390764    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97" event={"ID":"419b608e-bfa8-4d2b-bacd-08c286fb7707","Type":"ContainerStarted","Data":"8698545d4a79dded51e07962a78d7124c90fe8143c26dc272d56e3feb94ed846"}
Oct 30 19:42:24.392640 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:24.392614    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl" event={"ID":"a94cd52c-5a75-45c0-9669-587b56e8fc70","Type":"ContainerStarted","Data":"3c50196191af194a41564f3267c2843456b0ef62c3f8947335714d2130a91440"}
Oct 30 19:42:24.392757 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:24.392669    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl" event={"ID":"a94cd52c-5a75-45c0-9669-587b56e8fc70","Type":"ContainerStarted","Data":"085c785660f5c7806f01723b16281e849e119cb76f7dd54458cf8b126244d11c"}
Oct 30 19:42:24.392757 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:24.392698    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl" event={"ID":"a94cd52c-5a75-45c0-9669-587b56e8fc70","Type":"ContainerStarted","Data":"21b5386a6dae83c293ec44372355511c77f4f6b45f376989b85773bcbc034f3c"}
Oct 30 19:42:24.393839 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:24.393814    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t586q" event={"ID":"56bbeb6c-dce1-42dc-9291-615c39153a7d","Type":"ContainerStarted","Data":"f7a0c42e78195e1214da15d460376329d87d668b2a0e8568d6fd317c439b54c8"}
Oct 30 19:42:25.398623 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.398588    2573 generic.go:356] "Generic (PLEG): container finished" podID="56bbeb6c-dce1-42dc-9291-615c39153a7d" containerID="f4042ad646245e91d971a4165882870ec5260ee6f3f43f6fb10dc07470202097" exitCode=0
Oct 30 19:42:25.398992 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.398635    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t586q" event={"ID":"56bbeb6c-dce1-42dc-9291-615c39153a7d","Type":"ContainerDied","Data":"f4042ad646245e91d971a4165882870ec5260ee6f3f43f6fb10dc07470202097"}
Oct 30 19:42:25.522209 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.521443    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:42:25.533224 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.532882    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.602887 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.602841    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 19:42:25.603060 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.603016    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 19:42:25.603340 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.603301    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 19:42:25.636884 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.636842    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d79cb0a2-da98-4c41-a201-50755411d3c1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.637021 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.636953    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8b99\" (UniqueName: \"kubernetes.io/projected/d79cb0a2-da98-4c41-a201-50755411d3c1-kube-api-access-v8b99\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.637021 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.637004    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.637089 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.637033    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-config-volume\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.637089 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.637054    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.637167 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.637120    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.637213 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.637178    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d79cb0a2-da98-4c41-a201-50755411d3c1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.637250 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.637232    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d79cb0a2-da98-4c41-a201-50755411d3c1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.637292 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.637280    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-web-config\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.637336 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.637303    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.637366 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.637334    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.637405 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.637375    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d79cb0a2-da98-4c41-a201-50755411d3c1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.637467 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.637424    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d79cb0a2-da98-4c41-a201-50755411d3c1-config-out\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.719268 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.719224    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 19:42:25.719467 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.719274    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 19:42:25.719467 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.719324    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 19:42:25.719467 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.719395    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 19:42:25.719467 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.719405    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 19:42:25.719467 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.719420    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2zvp8\""
Oct 30 19:42:25.720210 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.720187    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 19:42:25.720926 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.720906    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:42:25.739744 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.739709    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.739893 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.739786    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d79cb0a2-da98-4c41-a201-50755411d3c1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.739893 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.739816    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d79cb0a2-da98-4c41-a201-50755411d3c1-config-out\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.739893 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.739852    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d79cb0a2-da98-4c41-a201-50755411d3c1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.740037 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.739890    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8b99\" (UniqueName: \"kubernetes.io/projected/d79cb0a2-da98-4c41-a201-50755411d3c1-kube-api-access-v8b99\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.740037 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.739958    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.740037 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.739996    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-config-volume\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.740037 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.740027    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.740216 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.740050    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.740216 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.740073    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d79cb0a2-da98-4c41-a201-50755411d3c1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.740216 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.740093    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d79cb0a2-da98-4c41-a201-50755411d3c1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.740216 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.740122    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-web-config\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.740216 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.740152    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.741806 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.741489    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d79cb0a2-da98-4c41-a201-50755411d3c1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.741806 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.741760    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d79cb0a2-da98-4c41-a201-50755411d3c1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.743085 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.743059    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d79cb0a2-da98-4c41-a201-50755411d3c1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.743590 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.743556    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.746106 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.746076    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d79cb0a2-da98-4c41-a201-50755411d3c1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.749836 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.749804    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d79cb0a2-da98-4c41-a201-50755411d3c1-config-out\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.749983 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.749958    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-config-volume\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.750477 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.750418    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-web-config\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.750477 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.750428    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.751009 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.750985    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.752727 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.752706    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.754801 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.754779    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.831751 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.831721    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8b99\" (UniqueName: \"kubernetes.io/projected/d79cb0a2-da98-4c41-a201-50755411d3c1-kube-api-access-v8b99\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.854431 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.854394    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:25.936990 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.936945    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"]
Oct 30 19:42:25.940570 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.940544    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:25.997319 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.997289    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 19:42:25.997319 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.997293    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 19:42:25.997507 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.997407    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 19:42:25.997507 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.997476    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7t1rbpjevshnn\""
Oct 30 19:42:25.997678 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.997535    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 19:42:25.997678 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.997553    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-v5m2b\""
Oct 30 19:42:25.997678 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:25.997639    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 19:42:26.008988 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.008963    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"]
Oct 30 19:42:26.041409 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.041385    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.041508 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.041422    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.041508 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.041494    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-grpc-tls\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.041583 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.041512    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25a55956-200b-4ca6-ad5e-050caf7116e4-tmp\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.041583 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.041563    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.041639 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.041622    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.041691 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.041676    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-tls\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.041778 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.041757    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25a55956-200b-4ca6-ad5e-050caf7116e4-metrics-client-ca\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.041810 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.041794    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbcq7\" (UniqueName: \"kubernetes.io/projected/25a55956-200b-4ca6-ad5e-050caf7116e4-kube-api-access-nbcq7\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.142348 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.142322    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-grpc-tls\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.142444 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.142354    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25a55956-200b-4ca6-ad5e-050caf7116e4-tmp\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.142444 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.142389    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.142733 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.142487    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.142733 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.142555    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-tls\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.142733 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.142593    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25a55956-200b-4ca6-ad5e-050caf7116e4-metrics-client-ca\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.142733 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.142622    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbcq7\" (UniqueName: \"kubernetes.io/projected/25a55956-200b-4ca6-ad5e-050caf7116e4-kube-api-access-nbcq7\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.142733 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.142667    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.142733 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.142697    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.142979 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.142839    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25a55956-200b-4ca6-ad5e-050caf7116e4-tmp\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.144063 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.144009    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25a55956-200b-4ca6-ad5e-050caf7116e4-metrics-client-ca\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.145586 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.145558    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.145586 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.145573    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-grpc-tls\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.146051 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.146012    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.146436 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.146406    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.146776 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.146756    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.147200 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.147184    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-tls\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.209644 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.209570    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbcq7\" (UniqueName: \"kubernetes.io/projected/25a55956-200b-4ca6-ad5e-050caf7116e4-kube-api-access-nbcq7\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.258701 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.258651    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:26.335336 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.335311    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:42:26.337537 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:42:26.337512    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79cb0a2_da98_4c41_a201_50755411d3c1.slice/crio-ca771da4e671d62f4d3e26f0a503559a4c02b532afec77ed8b75626b74cbe8ea WatchSource:0}: Error finding container ca771da4e671d62f4d3e26f0a503559a4c02b532afec77ed8b75626b74cbe8ea: Status 404 returned error can't find the container with id ca771da4e671d62f4d3e26f0a503559a4c02b532afec77ed8b75626b74cbe8ea
Oct 30 19:42:26.361662 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.361635    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-zk9rp"]
Oct 30 19:42:26.364109 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.364095    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-zk9rp"
Oct 30 19:42:26.407490 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.407439    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97" event={"ID":"419b608e-bfa8-4d2b-bacd-08c286fb7707","Type":"ContainerStarted","Data":"fa99e5ed615bb0a659fe5023242febe367bfeeb5475e3af4efa4a2a4233e29d0"}
Oct 30 19:42:26.407923 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.407542    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97" event={"ID":"419b608e-bfa8-4d2b-bacd-08c286fb7707","Type":"ContainerStarted","Data":"a628ecd0591988282d603a873436b94a06776d22b538cf9a3e51d9542abd3a5c"}
Oct 30 19:42:26.407923 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.407564    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97" event={"ID":"419b608e-bfa8-4d2b-bacd-08c286fb7707","Type":"ContainerStarted","Data":"34d042f610ce3aa38f8e74e8c72fee61085cc9c75307959bb23745c2c1aa81c9"}
Oct 30 19:42:26.409579 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.409555    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl" event={"ID":"a94cd52c-5a75-45c0-9669-587b56e8fc70","Type":"ContainerStarted","Data":"4152549f1db6ae689041b21f20bce86bc662ab91b08d54285adf0c4e547c1152"}
Oct 30 19:42:26.411687 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.411660    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t586q" event={"ID":"56bbeb6c-dce1-42dc-9291-615c39153a7d","Type":"ContainerStarted","Data":"0fa9784804e06ddf2b15b807190678987498557cbc7f85f8f7daa7371407bb3a"}
Oct 30 19:42:26.411771 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.411704    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t586q" event={"ID":"56bbeb6c-dce1-42dc-9291-615c39153a7d","Type":"ContainerStarted","Data":"58317b2433837e4dc9cbfe6000a1dd7a54e6a242b8aeb69c14cf050d8b2f516e"}
Oct 30 19:42:26.412823 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.412804    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d79cb0a2-da98-4c41-a201-50755411d3c1","Type":"ContainerStarted","Data":"ca771da4e671d62f4d3e26f0a503559a4c02b532afec77ed8b75626b74cbe8ea"}
Oct 30 19:42:26.422866 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.422845    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 19:42:26.433169 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.433143    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-zk9rp"]
Oct 30 19:42:26.524789 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.524711    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 19:42:26.546546 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.546508    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cf2adf5-da5b-4d63-a490-9e88f3598288-tmp\") pod \"downloads-6dd789c8b7-zk9rp\" (UID: \"7cf2adf5-da5b-4d63-a490-9e88f3598288\") " pod="openshift-console/downloads-6dd789c8b7-zk9rp"
Oct 30 19:42:26.546696 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.546579    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s45zp\" (UniqueName: \"kubernetes.io/projected/7cf2adf5-da5b-4d63-a490-9e88f3598288-kube-api-access-s45zp\") pod \"downloads-6dd789c8b7-zk9rp\" (UID: \"7cf2adf5-da5b-4d63-a490-9e88f3598288\") " pod="openshift-console/downloads-6dd789c8b7-zk9rp"
Oct 30 19:42:26.637322 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.637286    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"]
Oct 30 19:42:26.638644 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.638511    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97" podStartSLOduration=1.785190258 podStartE2EDuration="3.638497301s" totalImagesPullingTime="1.853307043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:26.637817807 +0000 UTC m=+58.982697295" watchObservedRunningTime="2026-10-30 19:42:26.638497301 +0000 UTC m=+58.983376790"
Oct 30 19:42:26.640987 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:42:26.640960    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a55956_200b_4ca6_ad5e_050caf7116e4.slice/crio-fa8572cc41c0e22b72a4e7b7b0683e571485522d6f14b5913769a8fd574a2beb WatchSource:0}: Error finding container fa8572cc41c0e22b72a4e7b7b0683e571485522d6f14b5913769a8fd574a2beb: Status 404 returned error can't find the container with id fa8572cc41c0e22b72a4e7b7b0683e571485522d6f14b5913769a8fd574a2beb
Oct 30 19:42:26.648065 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.648038    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cf2adf5-da5b-4d63-a490-9e88f3598288-tmp\") pod \"downloads-6dd789c8b7-zk9rp\" (UID: \"7cf2adf5-da5b-4d63-a490-9e88f3598288\") " pod="openshift-console/downloads-6dd789c8b7-zk9rp"
Oct 30 19:42:26.648214 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.648083    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s45zp\" (UniqueName: \"kubernetes.io/projected/7cf2adf5-da5b-4d63-a490-9e88f3598288-kube-api-access-s45zp\") pod \"downloads-6dd789c8b7-zk9rp\" (UID: \"7cf2adf5-da5b-4d63-a490-9e88f3598288\") " pod="openshift-console/downloads-6dd789c8b7-zk9rp"
Oct 30 19:42:26.648432 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.648410    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cf2adf5-da5b-4d63-a490-9e88f3598288-tmp\") pod \"downloads-6dd789c8b7-zk9rp\" (UID: \"7cf2adf5-da5b-4d63-a490-9e88f3598288\") " pod="openshift-console/downloads-6dd789c8b7-zk9rp"
Oct 30 19:42:26.718965 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.718917    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s45zp\" (UniqueName: \"kubernetes.io/projected/7cf2adf5-da5b-4d63-a490-9e88f3598288-kube-api-access-s45zp\") pod \"downloads-6dd789c8b7-zk9rp\" (UID: \"7cf2adf5-da5b-4d63-a490-9e88f3598288\") " pod="openshift-console/downloads-6dd789c8b7-zk9rp"
Oct 30 19:42:26.781252 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.781144    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-t586q" podStartSLOduration=2.307469169 podStartE2EDuration="3.78112735s" totalImagesPullingTime="1.399028779s" totalInitContainerRuntime="74.629402ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:42:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:26.780490034 +0000 UTC m=+59.125369524" watchObservedRunningTime="2026-10-30 19:42:26.78112735 +0000 UTC m=+59.126006837"
Oct 30 19:42:26.902845 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.902779    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl" podStartSLOduration=3.446009283 podStartE2EDuration="4.902760704s" totalImagesPullingTime="1.456751421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:26.901894176 +0000 UTC m=+59.246773664" watchObservedRunningTime="2026-10-30 19:42:26.902760704 +0000 UTC m=+59.247640192"
Oct 30 19:42:26.973389 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:26.973348    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-zk9rp"
Oct 30 19:42:27.348826 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:27.348791    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-zk9rp"]
Oct 30 19:42:27.352145 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:42:27.352119    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cf2adf5_da5b_4d63_a490_9e88f3598288.slice/crio-493d40a2457d2969980842246b67511aadca0feb4648ebac9bc8683bff5adc0c WatchSource:0}: Error finding container 493d40a2457d2969980842246b67511aadca0feb4648ebac9bc8683bff5adc0c: Status 404 returned error can't find the container with id 493d40a2457d2969980842246b67511aadca0feb4648ebac9bc8683bff5adc0c
Oct 30 19:42:27.417567 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:27.417480    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-zk9rp" event={"ID":"7cf2adf5-da5b-4d63-a490-9e88f3598288","Type":"ContainerStarted","Data":"493d40a2457d2969980842246b67511aadca0feb4648ebac9bc8683bff5adc0c"}
Oct 30 19:42:27.419037 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:27.418998    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl" event={"ID":"25a55956-200b-4ca6-ad5e-050caf7116e4","Type":"ContainerStarted","Data":"fa8572cc41c0e22b72a4e7b7b0683e571485522d6f14b5913769a8fd574a2beb"}
Oct 30 19:42:27.851877 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:27.851854    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-p527g"]
Oct 30 19:42:27.853669 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:27.853654    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p527g"
Oct 30 19:42:27.906507 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:27.906486    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-v9l6w\""
Oct 30 19:42:27.907389 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:27.907356    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 19:42:27.920463 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:27.920419    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-p527g"]
Oct 30 19:42:27.960853 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:27.960819    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6f405174-11f7-414c-a940-2a63c8989f3a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-p527g\" (UID: \"6f405174-11f7-414c-a940-2a63c8989f3a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p527g"
Oct 30 19:42:27.960994 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:27.960978    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgm8w\" (UniqueName: \"kubernetes.io/projected/6f405174-11f7-414c-a940-2a63c8989f3a-kube-api-access-dgm8w\") pod \"monitoring-plugin-77f4f76594-p527g\" (UID: \"6f405174-11f7-414c-a940-2a63c8989f3a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p527g"
Oct 30 19:42:28.061759 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.061677    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgm8w\" (UniqueName: \"kubernetes.io/projected/6f405174-11f7-414c-a940-2a63c8989f3a-kube-api-access-dgm8w\") pod \"monitoring-plugin-77f4f76594-p527g\" (UID: \"6f405174-11f7-414c-a940-2a63c8989f3a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p527g"
Oct 30 19:42:28.061759 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.061718    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6f405174-11f7-414c-a940-2a63c8989f3a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-p527g\" (UID: \"6f405174-11f7-414c-a940-2a63c8989f3a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p527g"
Oct 30 19:42:28.114766 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.114740    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 19:42:28.125292 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.125267    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6f405174-11f7-414c-a940-2a63c8989f3a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-p527g\" (UID: \"6f405174-11f7-414c-a940-2a63c8989f3a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p527g"
Oct 30 19:42:28.133998 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.133959    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgm8w\" (UniqueName: \"kubernetes.io/projected/6f405174-11f7-414c-a940-2a63c8989f3a-kube-api-access-dgm8w\") pod \"monitoring-plugin-77f4f76594-p527g\" (UID: \"6f405174-11f7-414c-a940-2a63c8989f3a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p527g"
Oct 30 19:42:28.188670 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.188638    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-8698dc55b4-pwqbt"]
Oct 30 19:42:28.191493 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.191465    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:28.245925 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.245897    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 19:42:28.245925 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.245923    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 19:42:28.246881 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.246861    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 19:42:28.247033 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.247012    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-tbtn8\""
Oct 30 19:42:28.247123 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.247062    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 19:42:28.247184 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.247168    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-f0ill0u6gh1v0\""
Oct 30 19:42:28.256982 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.256963    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-v9l6w\""
Oct 30 19:42:28.259075 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.259052    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8698dc55b4-pwqbt"]
Oct 30 19:42:28.262756 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.262735    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p527g"
Oct 30 19:42:28.363668 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.363634    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/154facaf-4163-41fb-a6d9-fb04a5d8e12e-secret-metrics-server-tls\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:28.363837 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.363682    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/154facaf-4163-41fb-a6d9-fb04a5d8e12e-audit-log\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:28.363901 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.363877    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/154facaf-4163-41fb-a6d9-fb04a5d8e12e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:28.363953 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.363918    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85gjj\" (UniqueName: \"kubernetes.io/projected/154facaf-4163-41fb-a6d9-fb04a5d8e12e-kube-api-access-85gjj\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:28.363995 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.363951    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/154facaf-4163-41fb-a6d9-fb04a5d8e12e-metrics-server-audit-profiles\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:28.364044 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.364022    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/154facaf-4163-41fb-a6d9-fb04a5d8e12e-client-ca-bundle\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:28.364152 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.364123    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/154facaf-4163-41fb-a6d9-fb04a5d8e12e-secret-metrics-server-client-certs\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:28.401923 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.401898    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:28.425083 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.425061    2573 generic.go:356] "Generic (PLEG): container finished" podID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerID="a9af2f1df48ee00d2017099c3228cbb04f43cef68b474005219ce36c44e86c10" exitCode=0
Oct 30 19:42:28.425476 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.425112    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d79cb0a2-da98-4c41-a201-50755411d3c1","Type":"ContainerDied","Data":"a9af2f1df48ee00d2017099c3228cbb04f43cef68b474005219ce36c44e86c10"}
Oct 30 19:42:28.464493 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.464466    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/154facaf-4163-41fb-a6d9-fb04a5d8e12e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:28.464605 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.464500    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85gjj\" (UniqueName: \"kubernetes.io/projected/154facaf-4163-41fb-a6d9-fb04a5d8e12e-kube-api-access-85gjj\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:28.464605 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.464525    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/154facaf-4163-41fb-a6d9-fb04a5d8e12e-metrics-server-audit-profiles\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:28.464605 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.464547    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/154facaf-4163-41fb-a6d9-fb04a5d8e12e-client-ca-bundle\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:28.464729 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.464614    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/154facaf-4163-41fb-a6d9-fb04a5d8e12e-secret-metrics-server-client-certs\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:28.464729 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.464663    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/154facaf-4163-41fb-a6d9-fb04a5d8e12e-secret-metrics-server-tls\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:28.464729 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.464683    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/154facaf-4163-41fb-a6d9-fb04a5d8e12e-audit-log\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:28.465069 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.465038    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/154facaf-4163-41fb-a6d9-fb04a5d8e12e-audit-log\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:28.465255 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.465222    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/154facaf-4163-41fb-a6d9-fb04a5d8e12e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:28.465823 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.465796    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/154facaf-4163-41fb-a6d9-fb04a5d8e12e-metrics-server-audit-profiles\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:28.467258 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.467236    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/154facaf-4163-41fb-a6d9-fb04a5d8e12e-client-ca-bundle\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:28.467734 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.467713    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/154facaf-4163-41fb-a6d9-fb04a5d8e12e-secret-metrics-server-tls\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:28.467781 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.467713    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/154facaf-4163-41fb-a6d9-fb04a5d8e12e-secret-metrics-server-client-certs\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:28.531893 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.531864    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85gjj\" (UniqueName: \"kubernetes.io/projected/154facaf-4163-41fb-a6d9-fb04a5d8e12e-kube-api-access-85gjj\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:28.675024 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.674937    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-p527g"]
Oct 30 19:42:28.678048 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:42:28.678012    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f405174_11f7_414c_a940_2a63c8989f3a.slice/crio-9e995309e7a18f11cf075a79bdc9a12bd1c7e2fb0b42bd91980500d0be2c43c2 WatchSource:0}: Error finding container 9e995309e7a18f11cf075a79bdc9a12bd1c7e2fb0b42bd91980500d0be2c43c2: Status 404 returned error can't find the container with id 9e995309e7a18f11cf075a79bdc9a12bd1c7e2fb0b42bd91980500d0be2c43c2
Oct 30 19:42:28.803645 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.803607    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:28.826747 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.826720    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-596848ccc9-tsm2b"]
Oct 30 19:42:28.830006 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.829951    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:28.885290 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.885263    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 19:42:28.885290 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.885263    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 19:42:28.885512 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.885298    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 19:42:28.885512 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.885274    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-vxj8j\""
Oct 30 19:42:28.885512 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.885306    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 19:42:28.886291 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.886241    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 19:42:28.939921 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.939851    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 19:42:28.940882 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.940859    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-596848ccc9-tsm2b"]
Oct 30 19:42:28.970703 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.970670    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91ef5bd9-9316-418a-a989-8557a38e0058-metrics-client-ca\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:28.970870 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.970753    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91ef5bd9-9316-418a-a989-8557a38e0058-serving-certs-ca-bundle\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:28.970940 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.970871    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91ef5bd9-9316-418a-a989-8557a38e0058-telemeter-trusted-ca-bundle\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:28.970993 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.970967    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91ef5bd9-9316-418a-a989-8557a38e0058-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:28.971046 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.970998    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/91ef5bd9-9316-418a-a989-8557a38e0058-secret-telemeter-client\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:28.971046 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.971023    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/91ef5bd9-9316-418a-a989-8557a38e0058-telemeter-client-tls\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:28.971141 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.971046    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/91ef5bd9-9316-418a-a989-8557a38e0058-federate-client-tls\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:28.971141 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:28.971068    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n9wq\" (UniqueName: \"kubernetes.io/projected/91ef5bd9-9316-418a-a989-8557a38e0058-kube-api-access-7n9wq\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:29.071916 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:29.071880    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/91ef5bd9-9316-418a-a989-8557a38e0058-secret-telemeter-client\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:29.071916 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:29.071918    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/91ef5bd9-9316-418a-a989-8557a38e0058-telemeter-client-tls\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:29.072152 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:29.072064    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/91ef5bd9-9316-418a-a989-8557a38e0058-federate-client-tls\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:29.072152 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:29.072119    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n9wq\" (UniqueName: \"kubernetes.io/projected/91ef5bd9-9316-418a-a989-8557a38e0058-kube-api-access-7n9wq\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:29.072249 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:29.072153    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91ef5bd9-9316-418a-a989-8557a38e0058-metrics-client-ca\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:29.072249 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:29.072219    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91ef5bd9-9316-418a-a989-8557a38e0058-serving-certs-ca-bundle\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:29.072339 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:29.072271    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91ef5bd9-9316-418a-a989-8557a38e0058-telemeter-trusted-ca-bundle\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:29.072396 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:29.072349    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91ef5bd9-9316-418a-a989-8557a38e0058-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:29.073168 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:29.073138    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91ef5bd9-9316-418a-a989-8557a38e0058-serving-certs-ca-bundle\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:29.073280 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:29.073177    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91ef5bd9-9316-418a-a989-8557a38e0058-telemeter-trusted-ca-bundle\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:29.073422 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:29.073358    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91ef5bd9-9316-418a-a989-8557a38e0058-metrics-client-ca\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:29.074639 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:29.074614    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/91ef5bd9-9316-418a-a989-8557a38e0058-telemeter-client-tls\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:29.074913 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:29.074887    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91ef5bd9-9316-418a-a989-8557a38e0058-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:29.075204 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:29.075183    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/91ef5bd9-9316-418a-a989-8557a38e0058-secret-telemeter-client\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:29.075443 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:29.075423    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/91ef5bd9-9316-418a-a989-8557a38e0058-federate-client-tls\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:29.135746 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:29.135708    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n9wq\" (UniqueName: \"kubernetes.io/projected/91ef5bd9-9316-418a-a989-8557a38e0058-kube-api-access-7n9wq\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:29.147499 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:29.147474    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:29.192432 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:29.192312    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8698dc55b4-pwqbt"]
Oct 30 19:42:29.198430 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:42:29.198200    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154facaf_4163_41fb_a6d9_fb04a5d8e12e.slice/crio-fda962d53dc8abc96dd0958cd718190bb56fbd1046f067a4cbb3d964bdbf1ddc WatchSource:0}: Error finding container fda962d53dc8abc96dd0958cd718190bb56fbd1046f067a4cbb3d964bdbf1ddc: Status 404 returned error can't find the container with id fda962d53dc8abc96dd0958cd718190bb56fbd1046f067a4cbb3d964bdbf1ddc
Oct 30 19:42:29.430469 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:29.430407    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p527g" event={"ID":"6f405174-11f7-414c-a940-2a63c8989f3a","Type":"ContainerStarted","Data":"9e995309e7a18f11cf075a79bdc9a12bd1c7e2fb0b42bd91980500d0be2c43c2"}
Oct 30 19:42:29.431956 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:29.431928    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt" event={"ID":"154facaf-4163-41fb-a6d9-fb04a5d8e12e","Type":"ContainerStarted","Data":"fda962d53dc8abc96dd0958cd718190bb56fbd1046f067a4cbb3d964bdbf1ddc"}
Oct 30 19:42:29.537210 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:29.537115    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-596848ccc9-tsm2b"]
Oct 30 19:42:29.540423 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:42:29.540390    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ef5bd9_9316_418a_a989_8557a38e0058.slice/crio-354a9a1161c97b185d2c234e450f09223f9a182b889c82b6a76401363a432d66 WatchSource:0}: Error finding container 354a9a1161c97b185d2c234e450f09223f9a182b889c82b6a76401363a432d66: Status 404 returned error can't find the container with id 354a9a1161c97b185d2c234e450f09223f9a182b889c82b6a76401363a432d66
Oct 30 19:42:30.440923 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:30.440882    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl" event={"ID":"25a55956-200b-4ca6-ad5e-050caf7116e4","Type":"ContainerStarted","Data":"a55abbbef7d10f7b7601b6f9cfbeb63763073871c88710a8dc00108bd5b84f6d"}
Oct 30 19:42:30.441300 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:30.440975    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl" event={"ID":"25a55956-200b-4ca6-ad5e-050caf7116e4","Type":"ContainerStarted","Data":"8528b4e5a4c2faa2665fec16c36930a314e83575ca4bc7d90c2f235381d0ff53"}
Oct 30 19:42:30.441300 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:30.441018    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl" event={"ID":"25a55956-200b-4ca6-ad5e-050caf7116e4","Type":"ContainerStarted","Data":"ecd32d57d277ff72047d492a35dfd3f1bd307289b6ba8dcfe5801fd13c8ed91e"}
Oct 30 19:42:30.443805 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:30.443778    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b" event={"ID":"91ef5bd9-9316-418a-a989-8557a38e0058","Type":"ContainerStarted","Data":"354a9a1161c97b185d2c234e450f09223f9a182b889c82b6a76401363a432d66"}
Oct 30 19:42:31.459502 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:31.456922    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d79cb0a2-da98-4c41-a201-50755411d3c1","Type":"ContainerStarted","Data":"054a4388d3ec6bc2e9471fa4f2964f309d0cf36ebdc908990a21850c43b6aca2"}
Oct 30 19:42:31.459502 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:31.457002    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d79cb0a2-da98-4c41-a201-50755411d3c1","Type":"ContainerStarted","Data":"d02c58ca04cf1da1a94cb78ca4de61b4414708f8412a6284d2ac074667676deb"}
Oct 30 19:42:31.459502 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:31.457043    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d79cb0a2-da98-4c41-a201-50755411d3c1","Type":"ContainerStarted","Data":"a53d819b93dfeb0f7311cdfbdfee03f619136b2796c1c299632ddb2fc2882fc2"}
Oct 30 19:42:31.459502 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:31.457080    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d79cb0a2-da98-4c41-a201-50755411d3c1","Type":"ContainerStarted","Data":"89cc8f3dfe90249695ac5f814cc9ef56147bad383ba9c49f3c4e8cfab4c83dad"}
Oct 30 19:42:31.459502 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:31.457116    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d79cb0a2-da98-4c41-a201-50755411d3c1","Type":"ContainerStarted","Data":"f1e0f2582e88ef4830f08b643b1a96719eee6f9522f5859cf09355864a55a779"}
Oct 30 19:42:32.388870 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:32.388839    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rjdlc"
Oct 30 19:42:32.465204 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:32.465171    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d79cb0a2-da98-4c41-a201-50755411d3c1","Type":"ContainerStarted","Data":"e05fe4e8501643275a673151999aea5e8e64bef7c88bb8ce70c18496da9f9bae"}
Oct 30 19:42:32.468415 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:32.468391    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl" event={"ID":"25a55956-200b-4ca6-ad5e-050caf7116e4","Type":"ContainerStarted","Data":"53373d6ff0c7f1b08023d797b64bc59ee9bd191d8fcd0a6bb21cafcfbadbeaf9"}
Oct 30 19:42:32.468553 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:32.468480    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl" event={"ID":"25a55956-200b-4ca6-ad5e-050caf7116e4","Type":"ContainerStarted","Data":"5ffb7228d9fc9ff25adb4af135f54b3d6ed00634ef0a38264c655c77db91fe60"}
Oct 30 19:42:32.468553 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:32.468521    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl" event={"ID":"25a55956-200b-4ca6-ad5e-050caf7116e4","Type":"ContainerStarted","Data":"162068d4d490b866f7c2775a9dfafcb2b43bc1314006db0821ddf5cc44e0a449"}
Oct 30 19:42:32.468621 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:32.468553    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:32.469966 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:32.469944    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p527g" event={"ID":"6f405174-11f7-414c-a940-2a63c8989f3a","Type":"ContainerStarted","Data":"73785faacec37b6c36b010234fbc823ce95f16a98a92ea1860220e08af1ed898"}
Oct 30 19:42:32.470139 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:32.470117    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p527g"
Oct 30 19:42:32.471556 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:32.471444    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt" event={"ID":"154facaf-4163-41fb-a6d9-fb04a5d8e12e","Type":"ContainerStarted","Data":"dc6ba4530160a8c7604be4375ed25c1785ab2b88b66eafe412914e83b2266022"}
Oct 30 19:42:32.475543 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:32.475494    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p527g"
Oct 30 19:42:32.631064 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:32.630640    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt" podStartSLOduration=2.187060672 podStartE2EDuration="4.630620968s" totalImagesPullingTime="2.443560296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:32.630581832 +0000 UTC m=+64.975461322" watchObservedRunningTime="2026-10-30 19:42:32.630620968 +0000 UTC m=+64.975500458"
Oct 30 19:42:32.765401 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:32.765304    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.115096763 podStartE2EDuration="7.765289314s" totalImagesPullingTime="4.636518974s" totalInitContainerRuntime="13.673577ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:32.764027436 +0000 UTC m=+65.108906925" watchObservedRunningTime="2026-10-30 19:42:32.765289314 +0000 UTC m=+65.110168799"
Oct 30 19:42:32.895345 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:32.895286    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl" podStartSLOduration=3.187617909 podStartE2EDuration="7.895266169s" totalImagesPullingTime="4.70764826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:32.893689047 +0000 UTC m=+65.238568534" watchObservedRunningTime="2026-10-30 19:42:32.895266169 +0000 UTC m=+65.240145660"
Oct 30 19:42:33.018152 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.018046    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p527g" podStartSLOduration=3.266813554 podStartE2EDuration="6.018032534s" totalImagesPullingTime="2.75121898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:33.016208885 +0000 UTC m=+65.361088372" watchObservedRunningTime="2026-10-30 19:42:33.018032534 +0000 UTC m=+65.362912022"
Oct 30 19:42:33.027843 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.027818    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:42:33.031747 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.031724    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.093832 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.093804    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 19:42:33.105053 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.105026    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:42:33.113086 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113049    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.113228 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113099    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-web-config\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.113228 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113137    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-config-out\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.113228 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113159    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.113228 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113182    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.113228 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113214    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-config\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.113498 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113262    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.113498 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113298    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.113498 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113332    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.113498 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113356    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.113498 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113378    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.113498 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113398    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 19:42:33.113498 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113405    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.113498 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113442    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.113848 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113358    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 19:42:33.113848 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113502    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret\") pod \"global-pull-secret-syncer-8t48c\" (UID: \"93cfc69d-646b-4910-972d-04c9076c6c3c\") " pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:42:33.113848 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113607    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.113848 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113621    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 19:42:33.113848 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113662    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 19:42:33.113848 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113670    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 19:42:33.113848 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113688    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs\") pod \"network-metrics-daemon-rj77q\" (UID: \"9cd48d14-cd97-4894-a43e-fa678a1833b3\") " pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:42:33.113848 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113703    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 19:42:33.113848 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113705    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 19:42:33.113848 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113628    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tcwj6\""
Oct 30 19:42:33.113848 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113705    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 19:42:33.113848 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113774    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 19:42:33.114154 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113854    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.114154 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113880    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kp8z\" (UniqueName: \"kubernetes.io/projected/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-kube-api-access-5kp8z\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.114154 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113902    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d17vqncs64e3\""
Oct 30 19:42:33.114154 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113905    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.114154 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.113994    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.114154 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.114020    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.114154 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.114050    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.165259 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.165225    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:42:33.166404 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.166386    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:42:33.177075 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.177042    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs\") pod \"network-metrics-daemon-rj77q\" (UID: \"9cd48d14-cd97-4894-a43e-fa678a1833b3\") " pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:42:33.177075 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.177074    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret\") pod \"global-pull-secret-syncer-8t48c\" (UID: \"93cfc69d-646b-4910-972d-04c9076c6c3c\") " pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:42:33.194511 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.194486    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 19:42:33.194642 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.194529    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 19:42:33.215500 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.215427    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.215664 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.215513    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kp8z\" (UniqueName: \"kubernetes.io/projected/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-kube-api-access-5kp8z\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.215664 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.215539    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.215664 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.215568    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.215664 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.215599    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.215664 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.215621    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.215664 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.215643    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.215664 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.215661    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-web-config\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.215922 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.215682    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-config-out\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.215922 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.215697    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.215922 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.215718    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.215922 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.215744    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-config\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.215922 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.215764    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2tsl\" (UniqueName: \"kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl\") pod \"network-check-target-m9t6w\" (UID: \"14e5d4c8-c53b-43b7-8301-804226ba1867\") " pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:42:33.215922 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.215786    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.215922 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.215805    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.215922 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.215822    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.215922 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.215840    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.215922 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.215855    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.215922 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.215883    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.215922 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.215910    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.216371 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.215928    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.216371 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.216134    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.216444 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.216393    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.217911 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.217727    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.218436 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.218141    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.219883 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.219854    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.219965 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.219912    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.220022 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.219968    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-config\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.220261 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.220230    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.220757 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.220330    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.221908 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.221886    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.223537 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.223515    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.223865 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.223834    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.224762 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.224746    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.225172 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.225148    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-config-out\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.225318 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.225310    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-web-config\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.225761 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.225712    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.227278 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.227257    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.227734 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.227704    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 19:42:33.238214 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.238182    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.245143 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.245106    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 19:42:33.252145 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.252115    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.270793 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.270717    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:42:33.277545 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.277520    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kp8z\" (UniqueName: \"kubernetes.io/projected/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-kube-api-access-5kp8z\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.295692 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.295662    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2j2tt\""
Oct 30 19:42:33.304991 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.304965    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:42:33.306260 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.306239    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j4jh7\""
Oct 30 19:42:33.315290 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.315265    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:42:33.330038 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.329765    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:42:33.341427 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.341398    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2tsl\" (UniqueName: \"kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl\") pod \"network-check-target-m9t6w\" (UID: \"14e5d4c8-c53b-43b7-8301-804226ba1867\") " pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:42:33.359659 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.359632    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:33.479498 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.478516    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b" event={"ID":"91ef5bd9-9316-418a-a989-8557a38e0058","Type":"ContainerStarted","Data":"97a123874a6d270bbc3c2cd9e444ee08084611a0743f162f98d9814da608cb52"}
Oct 30 19:42:33.479498 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.478587    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b" event={"ID":"91ef5bd9-9316-418a-a989-8557a38e0058","Type":"ContainerStarted","Data":"dfc014d0b9ed51f279d083677a3a1f7191fe6505fcb88bfc6cbfbc48485d874a"}
Oct 30 19:42:33.479498 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.478616    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b" event={"ID":"91ef5bd9-9316-418a-a989-8557a38e0058","Type":"ContainerStarted","Data":"2729a8b97e6fb65a281272b6b2f7c6e1e0f5711becbef25a168fa728332a2b5b"}
Oct 30 19:42:33.605075 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.605049    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-vlbmc\""
Oct 30 19:42:33.610284 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.610258    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:42:33.626745 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.626699    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b" podStartSLOduration=2.7624775379999997 podStartE2EDuration="5.626682837s" totalImagesPullingTime="2.864205299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:33.625754601 +0000 UTC m=+65.970634090" watchObservedRunningTime="2026-10-30 19:42:33.626682837 +0000 UTC m=+65.971562323"
Oct 30 19:42:33.738613 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.738584    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8t48c"]
Oct 30 19:42:33.740969 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:42:33.740943    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93cfc69d_646b_4910_972d_04c9076c6c3c.slice/crio-187e8ebc30af6f594a6687401273619fc110adf12b8e0113e7ee57cea71c5d2f WatchSource:0}: Error finding container 187e8ebc30af6f594a6687401273619fc110adf12b8e0113e7ee57cea71c5d2f: Status 404 returned error can't find the container with id 187e8ebc30af6f594a6687401273619fc110adf12b8e0113e7ee57cea71c5d2f
Oct 30 19:42:33.763235 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.763214    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rj77q"]
Oct 30 19:42:33.765065 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:42:33.765039    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd48d14_cd97_4894_a43e_fa678a1833b3.slice/crio-554c4b77f9d0a7a7c176dd39e730b582e2e378b60a0607e6b63b7120b2e28a4a WatchSource:0}: Error finding container 554c4b77f9d0a7a7c176dd39e730b582e2e378b60a0607e6b63b7120b2e28a4a: Status 404 returned error can't find the container with id 554c4b77f9d0a7a7c176dd39e730b582e2e378b60a0607e6b63b7120b2e28a4a
Oct 30 19:42:33.815146 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:33.815106    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:42:33.818387 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:42:33.818355    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2acb7dd_f543_4acd_91c5_aa4b48c2caf9.slice/crio-8bdd2a1db11610307f5842206c66764c9cc590f10218b9fa345ffb3c8fffd446 WatchSource:0}: Error finding container 8bdd2a1db11610307f5842206c66764c9cc590f10218b9fa345ffb3c8fffd446: Status 404 returned error can't find the container with id 8bdd2a1db11610307f5842206c66764c9cc590f10218b9fa345ffb3c8fffd446
Oct 30 19:42:34.003355 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:34.003320    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m9t6w"]
Oct 30 19:42:34.006802 ip-10-0-142-240 kubenswrapper[2573]: W1030 19:42:34.006774    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e5d4c8_c53b_43b7_8301_804226ba1867.slice/crio-bb7ae1edbb9e66dd1e6c25b9bbca1f35d9f0749e81bbc071838660ae3cf02ce0 WatchSource:0}: Error finding container bb7ae1edbb9e66dd1e6c25b9bbca1f35d9f0749e81bbc071838660ae3cf02ce0: Status 404 returned error can't find the container with id bb7ae1edbb9e66dd1e6c25b9bbca1f35d9f0749e81bbc071838660ae3cf02ce0
Oct 30 19:42:34.485285 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:34.485252    2573 generic.go:356] "Generic (PLEG): container finished" podID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerID="83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef" exitCode=0
Oct 30 19:42:34.486064 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:34.486040    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9","Type":"ContainerDied","Data":"83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef"}
Oct 30 19:42:34.486167 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:34.486113    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9","Type":"ContainerStarted","Data":"8bdd2a1db11610307f5842206c66764c9cc590f10218b9fa345ffb3c8fffd446"}
Oct 30 19:42:34.487846 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:34.487819    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rj77q" event={"ID":"9cd48d14-cd97-4894-a43e-fa678a1833b3","Type":"ContainerStarted","Data":"554c4b77f9d0a7a7c176dd39e730b582e2e378b60a0607e6b63b7120b2e28a4a"}
Oct 30 19:42:34.489055 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:34.489034    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m9t6w" event={"ID":"14e5d4c8-c53b-43b7-8301-804226ba1867","Type":"ContainerStarted","Data":"bb7ae1edbb9e66dd1e6c25b9bbca1f35d9f0749e81bbc071838660ae3cf02ce0"}
Oct 30 19:42:34.490121 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:34.490101    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8t48c" event={"ID":"93cfc69d-646b-4910-972d-04c9076c6c3c","Type":"ContainerStarted","Data":"187e8ebc30af6f594a6687401273619fc110adf12b8e0113e7ee57cea71c5d2f"}
Oct 30 19:42:36.504634 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:36.504600    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rj77q" event={"ID":"9cd48d14-cd97-4894-a43e-fa678a1833b3","Type":"ContainerStarted","Data":"3af46df32e1a8addd0064ad2c8a13a02b432a8c7d67d67115a3aadeb64a32a7a"}
Oct 30 19:42:36.505520 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:36.504671    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rj77q" event={"ID":"9cd48d14-cd97-4894-a43e-fa678a1833b3","Type":"ContainerStarted","Data":"260e8b9a4096f0141d2dd76a4c20c36d2c6a6983e9f7b28b4e84b28352a6cdb9"}
Oct 30 19:42:36.645589 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:36.645487    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rj77q" podStartSLOduration=66.755391704 podStartE2EDuration="1m8.645468361s" totalImagesPullingTime="1.890076657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:36.641977224 +0000 UTC m=+68.986856715" watchObservedRunningTime="2026-10-30 19:42:36.645468361 +0000 UTC m=+68.990347849"
Oct 30 19:42:38.486867 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:38.486836    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:39.521933 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:39.521856    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m9t6w" event={"ID":"14e5d4c8-c53b-43b7-8301-804226ba1867","Type":"ContainerStarted","Data":"9240b16dcde6c9e2d8f0926bc0aec30a0fbe65395a707ba0563ceab4b2006098"}
Oct 30 19:42:39.522644 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:39.522623    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:42:40.532294 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:40.532255    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9","Type":"ContainerStarted","Data":"7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2"}
Oct 30 19:42:40.532850 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:40.532338    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9","Type":"ContainerStarted","Data":"ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981"}
Oct 30 19:42:40.532850 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:40.532379    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9","Type":"ContainerStarted","Data":"b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf"}
Oct 30 19:42:40.532850 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:40.532408    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9","Type":"ContainerStarted","Data":"423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03"}
Oct 30 19:42:40.532850 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:40.532431    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9","Type":"ContainerStarted","Data":"ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce"}
Oct 30 19:42:40.532850 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:40.532484    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9","Type":"ContainerStarted","Data":"8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168"}
Oct 30 19:42:40.683066 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:40.683002    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-m9t6w" podStartSLOduration=67.508187064 podStartE2EDuration="1m12.682980785s" totalImagesPullingTime="5.174793721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:39.64834877 +0000 UTC m=+71.993228257" watchObservedRunningTime="2026-10-30 19:42:40.682980785 +0000 UTC m=+73.027860274"
Oct 30 19:42:41.194367 ip-10-0-142-240 kubenswrapper[2573]: I1030 19:42:41.194222    2573 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:42:41.194349 ip-10-0-142-240 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 19:42:41.216894 ip-10-0-142-240 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 19:42:41.217078 ip-10-0-142-240 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 19:42:41.217135 ip-10-0-142-240 systemd[1]: kubelet.service: Consumed 4.624s CPU time.
Oct 30 19:42:41.232188 ip-10-0-142-240 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:42:41.297137 ip-10-0-142-240 kubenswrapper[5958]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:42:41.297137 ip-10-0-142-240 kubenswrapper[5958]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:42:41.297137 ip-10-0-142-240 kubenswrapper[5958]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:42:41.297137 ip-10-0-142-240 kubenswrapper[5958]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:42:41.300733 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300636    5958 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:42:41.300733 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300728    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:42:41.300733 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300733    5958 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:42:41.300733 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300736    5958 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:42:41.300733 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300740    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:42:41.300733 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300743    5958 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:42:41.300977 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300746    5958 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:42:41.300977 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300749    5958 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:42:41.300977 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300752    5958 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:42:41.300977 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300754    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:42:41.300977 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300757    5958 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:42:41.300977 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300760    5958 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:42:41.300977 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300762    5958 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:42:41.300977 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300765    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:42:41.300977 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300768    5958 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:42:41.300977 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300771    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:42:41.300977 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300773    5958 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:42:41.300977 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300776    5958 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:42:41.300977 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300779    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:42:41.300977 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300781    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:42:41.300977 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300784    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:42:41.300977 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300787    5958 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:42:41.300977 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300789    5958 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:42:41.300977 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300792    5958 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:42:41.300977 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300794    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:42:41.300977 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300797    5958 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:42:41.301561 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300800    5958 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:42:41.301561 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300803    5958 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:42:41.301561 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300805    5958 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:42:41.301561 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300808    5958 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:42:41.301561 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300811    5958 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:42:41.301561 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300817    5958 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:42:41.301561 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300820    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:42:41.301561 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300823    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:42:41.301561 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300826    5958 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:42:41.301561 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300828    5958 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:42:41.301561 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300831    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:42:41.301561 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300834    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:42:41.301561 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300836    5958 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:42:41.301561 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300839    5958 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:42:41.301561 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300841    5958 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:42:41.301561 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300844    5958 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:42:41.301561 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300847    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:42:41.301561 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300850    5958 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:42:41.301561 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300852    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:42:41.302086 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300855    5958 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:42:41.302086 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300858    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:42:41.302086 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300860    5958 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:42:41.302086 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300863    5958 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:42:41.302086 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300865    5958 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:42:41.302086 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300868    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:42:41.302086 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300870    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:42:41.302086 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300873    5958 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:42:41.302086 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300875    5958 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:42:41.302086 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300878    5958 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:42:41.302086 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300881    5958 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:42:41.302086 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300884    5958 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:42:41.302086 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300886    5958 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:42:41.302086 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300890    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:42:41.302086 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300892    5958 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:42:41.302086 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300895    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:42:41.302086 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300897    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:42:41.302086 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300901    5958 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:42:41.302086 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300906    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:42:41.302576 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300909    5958 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:42:41.302576 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300913    5958 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:42:41.302576 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300916    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:42:41.302576 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300920    5958 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:42:41.302576 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300924    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:42:41.302576 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300927    5958 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:42:41.302576 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300930    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:42:41.302576 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300933    5958 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:42:41.302576 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300936    5958 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:42:41.302576 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300939    5958 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:42:41.302576 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300941    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:42:41.302576 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300945    5958 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:42:41.302576 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300947    5958 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:42:41.302576 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300950    5958 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:42:41.302576 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300953    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:42:41.302576 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300955    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:42:41.302576 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300958    5958 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:42:41.302576 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300960    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:42:41.302576 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300963    5958 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:42:41.303112 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300972    5958 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:42:41.303112 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300975    5958 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:42:41.303112 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300978    5958 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:42:41.303112 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300980    5958 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:42:41.303112 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300983    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:42:41.303112 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300985    5958 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:42:41.303112 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300988    5958 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:42:41.303112 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300992    5958 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:42:41.303112 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300995    5958 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:42:41.303112 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.300997    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:42:41.303112 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301000    5958 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:42:41.303112 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301003    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:42:41.303112 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301005    5958 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:42:41.303112 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301007    5958 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:42:41.303112 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301010    5958 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:42:41.303112 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301012    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:42:41.303112 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301015    5958 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:42:41.303112 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301017    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:42:41.303112 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301020    5958 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:42:41.303112 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301022    5958 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:42:41.303675 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301025    5958 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:42:41.303675 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301027    5958 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:42:41.303675 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301030    5958 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:42:41.303675 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301803    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:42:41.303675 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301812    5958 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:42:41.303675 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301816    5958 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:42:41.303675 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301821    5958 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:42:41.303675 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301824    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:42:41.303675 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301828    5958 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:42:41.303675 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301831    5958 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:42:41.303675 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301834    5958 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:42:41.303675 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301837    5958 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:42:41.303675 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301839    5958 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:42:41.303675 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301843    5958 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:42:41.303675 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301845    5958 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:42:41.303675 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301848    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:42:41.303675 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301851    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:42:41.303675 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301853    5958 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:42:41.303675 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301856    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:42:41.304157 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301859    5958 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:42:41.304157 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301861    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:42:41.304157 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301864    5958 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:42:41.304157 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301866    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:42:41.304157 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301869    5958 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:42:41.304157 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301871    5958 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:42:41.304157 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301874    5958 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:42:41.304157 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301877    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:42:41.304157 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301879    5958 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:42:41.304157 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301882    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:42:41.304157 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301884    5958 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:42:41.304157 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301887    5958 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:42:41.304157 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301890    5958 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:42:41.304157 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301892    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:42:41.304157 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301895    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:42:41.304157 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301900    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:42:41.304157 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301902    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:42:41.304157 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301905    5958 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:42:41.304157 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301908    5958 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:42:41.304157 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301911    5958 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:42:41.304667 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301913    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:42:41.304667 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301916    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:42:41.304667 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301918    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:42:41.304667 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301921    5958 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:42:41.304667 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301923    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:42:41.304667 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301926    5958 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:42:41.304667 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301928    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:42:41.304667 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301938    5958 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:42:41.304667 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301942    5958 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:42:41.304667 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301945    5958 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:42:41.304667 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301948    5958 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:42:41.304667 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301951    5958 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:42:41.304667 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301954    5958 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:42:41.304667 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301956    5958 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:42:41.304667 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301959    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:42:41.304667 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301962    5958 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:42:41.304667 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301964    5958 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:42:41.304667 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301967    5958 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:42:41.304667 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301969    5958 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:42:41.304667 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301971    5958 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:42:41.305198 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301974    5958 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:42:41.305198 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301976    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:42:41.305198 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301979    5958 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:42:41.305198 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301982    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:42:41.305198 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301985    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:42:41.305198 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301987    5958 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:42:41.305198 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301990    5958 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:42:41.305198 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301994    5958 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:42:41.305198 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301997    5958 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:42:41.305198 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.301999    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:42:41.305198 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302002    5958 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:42:41.305198 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302004    5958 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:42:41.305198 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302007    5958 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:42:41.305198 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302009    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:42:41.305198 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302012    5958 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:42:41.305198 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302014    5958 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:42:41.305198 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302017    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:42:41.305198 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302019    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:42:41.305198 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302022    5958 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:42:41.305669 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302025    5958 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:42:41.305669 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302028    5958 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:42:41.305669 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302031    5958 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:42:41.305669 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302035    5958 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:42:41.305669 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302039    5958 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:42:41.305669 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302044    5958 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:42:41.305669 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302048    5958 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:42:41.305669 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302052    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:42:41.305669 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302054    5958 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:42:41.305669 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302057    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:42:41.305669 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302059    5958 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:42:41.305669 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302062    5958 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:42:41.305669 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302064    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:42:41.305669 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302067    5958 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:42:41.305669 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302069    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:42:41.305669 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302071    5958 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:42:41.305669 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302074    5958 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:42:41.305669 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302077    5958 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:42:41.305669 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302079    5958 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:42:41.305669 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302082    5958 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:42:41.306188 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302085    5958 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:42:41.306188 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302088    5958 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:42:41.306188 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302091    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:42:41.306188 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302094    5958 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:42:41.306188 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302096    5958 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:42:41.306188 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302099    5958 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:42:41.306188 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302101    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:42:41.306188 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302104    5958 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:42:41.306188 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302106    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:42:41.306188 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302109    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:42:41.306188 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302113    5958 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:42:41.306188 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302628    5958 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:42:41.306188 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302636    5958 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:42:41.306188 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302641    5958 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:42:41.306188 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302644    5958 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:42:41.306188 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302647    5958 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:42:41.306188 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302650    5958 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:42:41.306188 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302654    5958 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:42:41.306188 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302657    5958 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:42:41.306714 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302660    5958 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:42:41.306714 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302662    5958 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:42:41.306714 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302666    5958 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:42:41.306714 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302670    5958 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:42:41.306714 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302673    5958 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:42:41.306714 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302677    5958 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:42:41.306714 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302680    5958 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:42:41.306714 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302682    5958 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:42:41.306714 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302685    5958 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:42:41.306714 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302687    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:42:41.306714 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302690    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:42:41.306714 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302692    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:42:41.306714 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302695    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:42:41.306714 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302698    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:42:41.306714 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302701    5958 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:42:41.306714 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302703    5958 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:42:41.306714 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302706    5958 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:42:41.306714 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302708    5958 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:42:41.306714 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302711    5958 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:42:41.307178 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302713    5958 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:42:41.307178 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302716    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:42:41.307178 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302718    5958 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:42:41.307178 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302721    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:42:41.307178 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302723    5958 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:42:41.307178 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302726    5958 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:42:41.307178 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302728    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:42:41.307178 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302730    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:42:41.307178 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302733    5958 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:42:41.307178 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302736    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:42:41.307178 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302738    5958 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:42:41.307178 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302743    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:42:41.307178 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302745    5958 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:42:41.307178 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302748    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:42:41.307178 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302750    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:42:41.307178 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302753    5958 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:42:41.307178 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302755    5958 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:42:41.307178 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302758    5958 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:42:41.307178 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302760    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:42:41.307178 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302768    5958 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:42:41.307700 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302770    5958 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:42:41.307700 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302773    5958 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:42:41.307700 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302775    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:42:41.307700 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302777    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:42:41.307700 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302780    5958 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:42:41.307700 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302782    5958 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:42:41.307700 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302785    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:42:41.307700 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302788    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:42:41.307700 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302791    5958 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:42:41.307700 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302794    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:42:41.307700 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302796    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:42:41.307700 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302799    5958 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:42:41.307700 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302801    5958 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:42:41.307700 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302803    5958 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:42:41.307700 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302806    5958 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:42:41.307700 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302809    5958 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:42:41.307700 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302811    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:42:41.307700 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302813    5958 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:42:41.307700 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302816    5958 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:42:41.308304 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302818    5958 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:42:41.308304 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302821    5958 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:42:41.308304 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302824    5958 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:42:41.308304 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302826    5958 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:42:41.308304 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302831    5958 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:42:41.308304 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302833    5958 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:42:41.308304 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302836    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:42:41.308304 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302838    5958 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:42:41.308304 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302841    5958 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:42:41.308304 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302844    5958 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:42:41.308304 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302846    5958 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:42:41.308304 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302848    5958 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:42:41.308304 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302851    5958 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:42:41.308304 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302853    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:42:41.308304 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302856    5958 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:42:41.308304 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302858    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:42:41.308304 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302861    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:42:41.308304 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302863    5958 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:42:41.308304 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302866    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:42:41.308304 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302868    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:42:41.308797 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302871    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:42:41.308797 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302873    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:42:41.308797 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302876    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:42:41.308797 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302878    5958 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:42:41.308797 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302881    5958 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:42:41.308797 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302883    5958 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:42:41.308797 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302886    5958 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:42:41.308797 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302888    5958 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:42:41.308797 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302891    5958 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:42:41.308797 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302893    5958 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:42:41.308797 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302897    5958 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:42:41.308797 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302899    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:42:41.308797 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302902    5958 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:42:41.308797 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302904    5958 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:42:41.308797 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302907    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:42:41.308797 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302911    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:42:41.308797 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302914    5958 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:42:41.308797 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302917    5958 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:42:41.308797 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302919    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:42:41.308797 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.302922    5958 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:42:41.309283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303016    5958 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:42:41.309283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303024    5958 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:42:41.309283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303033    5958 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:42:41.309283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303038    5958 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:42:41.309283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303043    5958 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:42:41.309283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303046    5958 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:42:41.309283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303058    5958 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:42:41.309283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303062    5958 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:42:41.309283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303065    5958 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:42:41.309283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303068    5958 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:42:41.309283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303071    5958 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:42:41.309283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303075    5958 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:42:41.309283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303078    5958 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:42:41.309283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303080    5958 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:42:41.309283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303083    5958 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:42:41.309283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303086    5958 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:42:41.309283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303089    5958 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:42:41.309283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303092    5958 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:42:41.309283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303096    5958 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:42:41.309283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303098    5958 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:42:41.309283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303101    5958 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:42:41.309283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303104    5958 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:42:41.309283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303109    5958 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:42:41.309283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303113    5958 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:42:41.309874 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303116    5958 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:42:41.309874 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303119    5958 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:42:41.309874 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303123    5958 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:42:41.309874 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303126    5958 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:42:41.309874 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303130    5958 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:42:41.309874 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303133    5958 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:42:41.309874 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303136    5958 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:42:41.309874 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303139    5958 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:42:41.309874 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303143    5958 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:42:41.309874 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303147    5958 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:42:41.309874 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303149    5958 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:42:41.309874 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303152    5958 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:42:41.309874 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303155    5958 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:42:41.309874 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303158    5958 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:42:41.309874 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303163    5958 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:42:41.309874 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303166    5958 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:42:41.309874 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303169    5958 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:42:41.309874 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303171    5958 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:42:41.309874 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303175    5958 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:42:41.309874 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303179    5958 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:42:41.309874 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303182    5958 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:42:41.309874 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303184    5958 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:42:41.309874 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303187    5958 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:42:41.309874 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303190    5958 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:42:41.309874 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303193    5958 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:42:41.310494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303196    5958 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:42:41.310494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303199    5958 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:42:41.310494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303202    5958 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:42:41.310494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303205    5958 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:42:41.310494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303207    5958 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:42:41.310494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303213    5958 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:42:41.310494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303216    5958 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:42:41.310494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303219    5958 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:42:41.310494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303222    5958 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:42:41.310494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303225    5958 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:42:41.310494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303228    5958 flags.go:64] FLAG: --help="false"
Oct 30 19:42:41.310494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303231    5958 flags.go:64] FLAG: --hostname-override="ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:42:41.310494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303234    5958 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:42:41.310494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303237    5958 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:42:41.310494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303240    5958 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:42:41.310494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303244    5958 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:42:41.310494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303247    5958 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:42:41.310494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303250    5958 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:42:41.310494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303252    5958 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:42:41.310494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303255    5958 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:42:41.310494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303258    5958 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:42:41.310494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303261    5958 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:42:41.310494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303264    5958 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:42:41.311089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303267    5958 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:42:41.311089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303270    5958 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:42:41.311089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303272    5958 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:42:41.311089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303276    5958 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:42:41.311089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303278    5958 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:42:41.311089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303281    5958 flags.go:64] FLAG: --lock-file=""
Oct 30 19:42:41.311089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303284    5958 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:42:41.311089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303287    5958 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:42:41.311089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303290    5958 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:42:41.311089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303296    5958 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:42:41.311089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303299    5958 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:42:41.311089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303302    5958 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:42:41.311089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303304    5958 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:42:41.311089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303307    5958 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:42:41.311089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303311    5958 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:42:41.311089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303314    5958 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:42:41.311089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303317    5958 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:42:41.311089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303321    5958 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:42:41.311089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303324    5958 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:42:41.311089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303328    5958 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:42:41.311089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303331    5958 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:42:41.311089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303342    5958 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:42:41.311089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303347    5958 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:42:41.311089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303350    5958 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:42:41.311089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303352    5958 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303355    5958 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303358    5958 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303365    5958 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303368    5958 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303371    5958 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303374    5958 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303377    5958 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303379    5958 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303382    5958 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303385    5958 flags.go:64] FLAG: --port="10250"
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303388    5958 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303391    5958 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0901a26e1f416de69"
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303394    5958 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303396    5958 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303399    5958 flags.go:64] FLAG: --register-node="true"
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303402    5958 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303406    5958 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303409    5958 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303411    5958 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303414    5958 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303418    5958 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303424    5958 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303427    5958 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303430    5958 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303433    5958 flags.go:64] FLAG: --runonce="false"
Oct 30 19:42:41.311710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303435    5958 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:42:41.312338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303438    5958 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:42:41.312338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303441    5958 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:42:41.312338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303444    5958 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:42:41.312338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303463    5958 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:42:41.312338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303469    5958 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:42:41.312338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303473    5958 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:42:41.312338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303477    5958 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:42:41.312338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303480    5958 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:42:41.312338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303483    5958 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:42:41.312338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303485    5958 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:42:41.312338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303488    5958 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:42:41.312338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303491    5958 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:42:41.312338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303494    5958 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:42:41.312338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303497    5958 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:42:41.312338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303502    5958 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:42:41.312338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303504    5958 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:42:41.312338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303507    5958 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:42:41.312338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303511    5958 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:42:41.312338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303514    5958 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:42:41.312338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303517    5958 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:42:41.312338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303521    5958 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:42:41.312338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303523    5958 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:42:41.312338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303526    5958 flags.go:64] FLAG: --v="2"
Oct 30 19:42:41.312338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303530    5958 flags.go:64] FLAG: --version="false"
Oct 30 19:42:41.312338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303535    5958 flags.go:64] FLAG: --vmodule=""
Oct 30 19:42:41.312996 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303538    5958 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:42:41.312996 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.303542    5958 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:42:41.312996 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303693    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:42:41.312996 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303698    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:42:41.312996 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303701    5958 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:42:41.312996 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303704    5958 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:42:41.312996 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303706    5958 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:42:41.312996 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303709    5958 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:42:41.312996 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303711    5958 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:42:41.312996 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303714    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:42:41.312996 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303717    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:42:41.312996 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303720    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:42:41.312996 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303723    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:42:41.312996 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303726    5958 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:42:41.312996 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303729    5958 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:42:41.312996 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303731    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:42:41.312996 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303734    5958 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:42:41.312996 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303736    5958 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:42:41.312996 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303739    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:42:41.313795 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303741    5958 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:42:41.313795 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303744    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:42:41.313795 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303746    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:42:41.313795 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303749    5958 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:42:41.313795 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303751    5958 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:42:41.313795 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303754    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:42:41.313795 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303756    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:42:41.313795 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303759    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:42:41.313795 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303762    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:42:41.313795 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303765    5958 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:42:41.313795 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303769    5958 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:42:41.313795 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303772    5958 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:42:41.313795 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303775    5958 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:42:41.313795 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303778    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:42:41.313795 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303781    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:42:41.313795 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303785    5958 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:42:41.313795 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303788    5958 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:42:41.313795 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303790    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:42:41.313795 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303793    5958 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:42:41.313795 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303795    5958 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:42:41.314562 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303798    5958 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:42:41.314562 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303801    5958 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:42:41.314562 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303803    5958 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:42:41.314562 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303806    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:42:41.314562 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303809    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:42:41.314562 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303811    5958 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:42:41.314562 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303814    5958 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:42:41.314562 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303817    5958 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:42:41.314562 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303819    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:42:41.314562 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303822    5958 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:42:41.314562 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303824    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:42:41.314562 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303827    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:42:41.314562 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303829    5958 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:42:41.314562 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303832    5958 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:42:41.314562 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303834    5958 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:42:41.314562 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303837    5958 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:42:41.314562 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303839    5958 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:42:41.314562 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303842    5958 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:42:41.314562 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303844    5958 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:42:41.314562 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303846    5958 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:42:41.315115 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303850    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:42:41.315115 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303853    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:42:41.315115 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303855    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:42:41.315115 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303858    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:42:41.315115 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303860    5958 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:42:41.315115 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303863    5958 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:42:41.315115 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303865    5958 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:42:41.315115 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303869    5958 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:42:41.315115 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303872    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:42:41.315115 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303874    5958 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:42:41.315115 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303877    5958 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:42:41.315115 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303880    5958 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:42:41.315115 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303882    5958 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:42:41.315115 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303885    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:42:41.315115 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303887    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:42:41.315115 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303890    5958 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:42:41.315115 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303894    5958 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:42:41.315115 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303897    5958 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:42:41.315115 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303900    5958 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:42:41.315709 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303903    5958 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:42:41.315709 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303906    5958 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:42:41.315709 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303909    5958 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:42:41.315709 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303911    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:42:41.315709 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303914    5958 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:42:41.315709 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303916    5958 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:42:41.315709 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303919    5958 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:42:41.315709 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303921    5958 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:42:41.315709 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303924    5958 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:42:41.315709 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303926    5958 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:42:41.315709 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303929    5958 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:42:41.315709 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303931    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:42:41.315709 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303934    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:42:41.315709 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303938    5958 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:42:41.315709 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303940    5958 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:42:41.315709 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303943    5958 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:42:41.315709 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303945    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:42:41.315709 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303948    5958 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:42:41.315709 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303951    5958 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:42:41.315709 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303953    5958 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:42:41.316587 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303957    5958 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:42:41.316587 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303960    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:42:41.316587 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303962    5958 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:42:41.316587 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303965    5958 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:42:41.316587 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303967    5958 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:42:41.316587 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303970    5958 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:42:41.316587 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303972    5958 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:42:41.316587 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303975    5958 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:42:41.316587 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303978    5958 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:42:41.316587 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.303980    5958 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:42:41.316587 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.304006    5958 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:42:41.316587 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304048    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:42:41.316587 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304053    5958 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:42:41.316587 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304056    5958 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:42:41.316587 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304059    5958 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:42:41.317249 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304062    5958 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:42:41.317249 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304065    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:42:41.317249 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304067    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:42:41.317249 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304070    5958 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:42:41.317249 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304074    5958 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:42:41.317249 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304078    5958 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:42:41.317249 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304081    5958 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:42:41.317249 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304085    5958 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:42:41.317249 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304088    5958 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:42:41.317249 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304090    5958 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:42:41.317249 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304094    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:42:41.317249 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304097    5958 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:42:41.317249 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304099    5958 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:42:41.317249 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304102    5958 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:42:41.317249 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304104    5958 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:42:41.317249 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304107    5958 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:42:41.317249 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304109    5958 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:42:41.317249 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304112    5958 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:42:41.317249 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304115    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:42:41.318031 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304118    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:42:41.318031 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304121    5958 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:42:41.318031 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304123    5958 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:42:41.318031 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304126    5958 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:42:41.318031 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304128    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:42:41.318031 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304131    5958 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:42:41.318031 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304133    5958 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:42:41.318031 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304136    5958 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:42:41.318031 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304138    5958 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:42:41.318031 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304141    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:42:41.318031 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304143    5958 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:42:41.318031 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304146    5958 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:42:41.318031 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304149    5958 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:42:41.318031 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304151    5958 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:42:41.318031 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304154    5958 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:42:41.318031 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304156    5958 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:42:41.318031 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304159    5958 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:42:41.318031 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304162    5958 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:42:41.318031 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304164    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:42:41.318676 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304167    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:42:41.318676 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304170    5958 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:42:41.318676 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304172    5958 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:42:41.318676 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304175    5958 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:42:41.318676 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304179    5958 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:42:41.318676 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304181    5958 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:42:41.318676 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304184    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:42:41.318676 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304186    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:42:41.318676 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304188    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:42:41.318676 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304191    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:42:41.318676 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304194    5958 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:42:41.318676 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304196    5958 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:42:41.318676 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304201    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:42:41.318676 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304203    5958 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:42:41.318676 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304206    5958 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:42:41.318676 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304208    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:42:41.318676 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304211    5958 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:42:41.318676 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304214    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:42:41.318676 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304216    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:42:41.319127 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304219    5958 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:42:41.319127 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304221    5958 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:42:41.319127 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304224    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:42:41.319127 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304226    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:42:41.319127 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304229    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:42:41.319127 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304231    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:42:41.319127 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304234    5958 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:42:41.319127 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304238    5958 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:42:41.319127 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304242    5958 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:42:41.319127 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304245    5958 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:42:41.319127 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304248    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:42:41.319127 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304250    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:42:41.319127 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304253    5958 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:42:41.319127 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304255    5958 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:42:41.319127 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304258    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:42:41.319127 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304260    5958 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:42:41.319127 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304263    5958 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:42:41.319127 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304267    5958 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:42:41.319127 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304269    5958 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:42:41.319127 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304273    5958 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:42:41.319626 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304275    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:42:41.319626 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304278    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:42:41.319626 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304281    5958 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:42:41.319626 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304283    5958 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:42:41.319626 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304286    5958 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:42:41.319626 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304290    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:42:41.319626 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304293    5958 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:42:41.319626 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304295    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:42:41.319626 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304298    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:42:41.319626 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304300    5958 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:42:41.319626 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304303    5958 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:42:41.319626 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304306    5958 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:42:41.319626 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304308    5958 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:42:41.319626 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304311    5958 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:42:41.319626 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304313    5958 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:42:41.319626 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304316    5958 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:42:41.319626 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304319    5958 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:42:41.319626 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304322    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:42:41.319626 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304324    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:42:41.319626 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304327    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:42:41.320093 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304330    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:42:41.320093 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304332    5958 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:42:41.320093 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304335    5958 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:42:41.320093 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304337    5958 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:42:41.320093 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.304340    5958 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:42:41.320093 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.308688    5958 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:42:41.320093 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.308704    5958 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:42:41.320093 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.308721    5958 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:42:41.320093 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.308726    5958 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:42:41.320093 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308786    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:42:41.320093 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308791    5958 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:42:41.320093 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308795    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:42:41.320093 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308799    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:42:41.320093 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308802    5958 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:42:41.320093 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308805    5958 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:42:41.320093 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308808    5958 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:42:41.320093 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308811    5958 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:42:41.320093 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308814    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:42:41.320093 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308817    5958 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:42:41.320564 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308819    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:42:41.320564 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308822    5958 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:42:41.320564 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308824    5958 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:42:41.320564 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308827    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:42:41.320564 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308829    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:42:41.320564 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308832    5958 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:42:41.320564 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308834    5958 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:42:41.320564 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308837    5958 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:42:41.320564 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308840    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:42:41.320564 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308842    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:42:41.320564 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308845    5958 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:42:41.320564 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308848    5958 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:42:41.320564 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308851    5958 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:42:41.320564 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308859    5958 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:42:41.320564 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308862    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:42:41.320564 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308864    5958 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:42:41.320564 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308867    5958 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:42:41.320564 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308869    5958 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:42:41.320564 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308872    5958 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:42:41.320564 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308875    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:42:41.321045 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308877    5958 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:42:41.321045 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308880    5958 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:42:41.321045 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308883    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:42:41.321045 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308885    5958 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:42:41.321045 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308887    5958 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:42:41.321045 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308890    5958 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:42:41.321045 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308892    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:42:41.321045 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308896    5958 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:42:41.321045 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308900    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:42:41.321045 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308903    5958 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:42:41.321045 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308905    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:42:41.321045 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308908    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:42:41.321045 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308911    5958 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:42:41.321045 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308913    5958 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:42:41.321045 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308916    5958 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:42:41.321045 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308918    5958 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:42:41.321045 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308921    5958 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:42:41.321045 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308924    5958 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:42:41.321045 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308926    5958 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:42:41.321521 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308929    5958 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:42:41.321521 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308932    5958 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:42:41.321521 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308934    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:42:41.321521 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308937    5958 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:42:41.321521 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308940    5958 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:42:41.321521 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308942    5958 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:42:41.321521 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308945    5958 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:42:41.321521 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308948    5958 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:42:41.321521 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308951    5958 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:42:41.321521 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308955    5958 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:42:41.321521 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308958    5958 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:42:41.321521 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308961    5958 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:42:41.321521 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308964    5958 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:42:41.321521 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308966    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:42:41.321521 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308969    5958 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:42:41.321521 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308971    5958 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:42:41.321521 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308974    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:42:41.321521 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308976    5958 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:42:41.321521 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308978    5958 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:42:41.321978 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308981    5958 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:42:41.321978 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308983    5958 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:42:41.321978 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308985    5958 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:42:41.321978 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308988    5958 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:42:41.321978 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308990    5958 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:42:41.321978 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308993    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:42:41.321978 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308995    5958 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:42:41.321978 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.308998    5958 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:42:41.321978 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309000    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:42:41.321978 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309003    5958 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:42:41.321978 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309005    5958 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:42:41.321978 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309007    5958 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:42:41.321978 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309010    5958 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:42:41.321978 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309012    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:42:41.321978 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309015    5958 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:42:41.321978 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309017    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:42:41.321978 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309020    5958 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:42:41.321978 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309022    5958 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:42:41.321978 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309025    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:42:41.321978 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309027    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:42:41.322620 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309030    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:42:41.322620 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309033    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:42:41.322620 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309035    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:42:41.322620 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309038    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:42:41.322620 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309040    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:42:41.322620 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309043    5958 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:42:41.322620 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309046    5958 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:42:41.322620 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309048    5958 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:42:41.322620 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309050    5958 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:42:41.322620 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309053    5958 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:42:41.322620 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309055    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:42:41.322620 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309058    5958 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:42:41.322620 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309060    5958 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:42:41.322620 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309062    5958 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:42:41.322620 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309065    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:42:41.322620 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309067    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:42:41.322620 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309070    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:42:41.322620 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309072    5958 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:42:41.323055 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.309098    5958 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:42:41.323055 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309237    5958 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:42:41.323055 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309242    5958 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:42:41.323055 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309245    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:42:41.323055 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309248    5958 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:42:41.323055 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309251    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:42:41.323055 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309253    5958 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:42:41.323055 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309256    5958 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:42:41.323055 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309258    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:42:41.323055 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309261    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:42:41.323055 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309263    5958 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:42:41.323055 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309266    5958 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:42:41.323055 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309268    5958 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:42:41.323055 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309271    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:42:41.323055 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309273    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:42:41.323055 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309276    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:42:41.323534 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309278    5958 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:42:41.323534 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309281    5958 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:42:41.323534 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309284    5958 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:42:41.323534 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309287    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:42:41.323534 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309289    5958 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:42:41.323534 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309292    5958 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:42:41.323534 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309295    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:42:41.323534 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309297    5958 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:42:41.323534 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309300    5958 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:42:41.323534 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309303    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:42:41.323534 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309305    5958 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:42:41.323534 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309307    5958 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:42:41.323534 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309310    5958 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:42:41.323534 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309313    5958 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:42:41.323534 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309315    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:42:41.323534 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309317    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:42:41.323534 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309321    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:42:41.323534 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309323    5958 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:42:41.323534 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309326    5958 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:42:41.323534 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309328    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:42:41.324028 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309330    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:42:41.324028 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309333    5958 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:42:41.324028 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309335    5958 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:42:41.324028 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309337    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:42:41.324028 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309340    5958 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:42:41.324028 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309342    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:42:41.324028 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309345    5958 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:42:41.324028 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309348    5958 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:42:41.324028 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309352    5958 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:42:41.324028 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309355    5958 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:42:41.324028 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309359    5958 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:42:41.324028 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309361    5958 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:42:41.324028 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309365    5958 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:42:41.324028 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309369    5958 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:42:41.324028 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309372    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:42:41.324028 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309374    5958 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:42:41.324028 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309377    5958 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:42:41.324028 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309379    5958 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:42:41.324480 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309382    5958 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:42:41.324480 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309384    5958 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:42:41.324480 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309386    5958 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:42:41.324480 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309389    5958 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:42:41.324480 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309391    5958 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:42:41.324480 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309393    5958 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:42:41.324480 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309396    5958 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:42:41.324480 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309398    5958 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:42:41.324480 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309401    5958 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:42:41.324480 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309403    5958 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:42:41.324480 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309406    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:42:41.324480 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309408    5958 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:42:41.324480 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309411    5958 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:42:41.324480 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309413    5958 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:42:41.324480 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309415    5958 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:42:41.324480 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309418    5958 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:42:41.324480 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309420    5958 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:42:41.324480 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309422    5958 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:42:41.324480 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309425    5958 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:42:41.324480 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309427    5958 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:42:41.324480 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309430    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:42:41.324972 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309432    5958 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:42:41.324972 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309435    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:42:41.324972 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309437    5958 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:42:41.324972 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309439    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:42:41.324972 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309442    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:42:41.324972 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309444    5958 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:42:41.324972 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309463    5958 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:42:41.324972 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309468    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:42:41.324972 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309471    5958 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:42:41.324972 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309473    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:42:41.324972 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309476    5958 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:42:41.324972 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309479    5958 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:42:41.324972 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309481    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:42:41.324972 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309484    5958 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:42:41.324972 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309486    5958 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:42:41.324972 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309489    5958 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:42:41.324972 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309491    5958 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:42:41.324972 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309493    5958 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:42:41.324972 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309500    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:42:41.325474 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309503    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:42:41.325474 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309505    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:42:41.325474 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309508    5958 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:42:41.325474 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309510    5958 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:42:41.325474 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309513    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:42:41.325474 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309515    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:42:41.325474 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309518    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:42:41.325474 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309520    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:42:41.325474 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309523    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:42:41.325474 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309525    5958 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:42:41.325474 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309528    5958 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:42:41.325474 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309530    5958 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:42:41.325474 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309532    5958 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:42:41.325474 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.309555    5958 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:42:41.325474 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309588    5958 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:42:41.325474 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309592    5958 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:42:41.325873 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309595    5958 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:42:41.325873 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309597    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:42:41.325873 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309600    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:42:41.325873 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309603    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:42:41.325873 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309605    5958 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:42:41.325873 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309609    5958 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:42:41.325873 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309611    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:42:41.325873 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309614    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:42:41.325873 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309617    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:42:41.325873 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309619    5958 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:42:41.325873 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309621    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:42:41.325873 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309624    5958 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:42:41.325873 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309626    5958 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:42:41.325873 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309628    5958 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:42:41.325873 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309631    5958 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:42:41.325873 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309633    5958 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:42:41.325873 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309637    5958 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:42:41.325873 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309640    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:42:41.325873 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309643    5958 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:42:41.326396 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309645    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:42:41.326396 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309647    5958 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:42:41.326396 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309650    5958 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:42:41.326396 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309652    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:42:41.326396 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309655    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:42:41.326396 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309657    5958 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:42:41.326396 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309660    5958 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:42:41.326396 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309662    5958 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:42:41.326396 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309665    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:42:41.326396 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309667    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:42:41.326396 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309670    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:42:41.326396 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309672    5958 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:42:41.326396 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309675    5958 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:42:41.326396 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309677    5958 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:42:41.326396 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309680    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:42:41.326396 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309683    5958 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:42:41.326396 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309685    5958 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:42:41.326396 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309689    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:42:41.326396 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309691    5958 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:42:41.326396 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309695    5958 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:42:41.326917 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309697    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:42:41.326917 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309700    5958 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:42:41.326917 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309702    5958 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:42:41.326917 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309704    5958 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:42:41.326917 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309707    5958 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:42:41.326917 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309710    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:42:41.326917 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309712    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:42:41.326917 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309715    5958 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:42:41.326917 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309718    5958 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:42:41.326917 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309722    5958 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:42:41.326917 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309724    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:42:41.326917 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309727    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:42:41.326917 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309729    5958 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:42:41.326917 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309732    5958 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:42:41.326917 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309734    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:42:41.326917 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309737    5958 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:42:41.326917 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309739    5958 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:42:41.326917 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309742    5958 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:42:41.326917 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309746    5958 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:42:41.327427 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309749    5958 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:42:41.327427 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309752    5958 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:42:41.327427 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309755    5958 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:42:41.327427 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309757    5958 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:42:41.327427 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309762    5958 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:42:41.327427 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309766    5958 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:42:41.327427 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309769    5958 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:42:41.327427 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309773    5958 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:42:41.327427 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309776    5958 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:42:41.327427 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309779    5958 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:42:41.327427 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309783    5958 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:42:41.327427 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309785    5958 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:42:41.327427 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309788    5958 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:42:41.327427 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309791    5958 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:42:41.327427 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309793    5958 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:42:41.327427 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309796    5958 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:42:41.327427 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309798    5958 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:42:41.327427 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309801    5958 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:42:41.327427 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309804    5958 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:42:41.327932 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309806    5958 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:42:41.327932 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309809    5958 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:42:41.327932 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309811    5958 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:42:41.327932 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309815    5958 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:42:41.327932 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309817    5958 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:42:41.327932 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309820    5958 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:42:41.327932 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309822    5958 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:42:41.327932 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309824    5958 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:42:41.327932 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309827    5958 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:42:41.327932 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309829    5958 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:42:41.327932 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309832    5958 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:42:41.327932 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309834    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:42:41.327932 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309837    5958 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:42:41.327932 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309839    5958 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:42:41.327932 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309842    5958 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:42:41.327932 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309844    5958 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:42:41.327932 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309847    5958 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:42:41.327932 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309849    5958 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:42:41.327932 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309853    5958 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:42:41.327932 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309856    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:42:41.328443 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309858    5958 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:42:41.328443 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309861    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:42:41.328443 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309863    5958 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:42:41.328443 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309867    5958 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:42:41.328443 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309869    5958 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:42:41.328443 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309872    5958 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:42:41.328443 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:42:41.309874    5958 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:42:41.328443 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.309925    5958 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:42:41.328443 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.310939    5958 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 19:42:41.328443 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.310999    5958 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 19:42:41.328443 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.311174    5958 server.go:1014] "Starting client certificate rotation"
Oct 30 19:42:41.328443 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.311306    5958 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:42:41.328443 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.311361    5958 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:36:28 +0000 UTC" deadline="2028-08-01 13:16:12.569836162 +0000 UTC"
Oct 30 19:42:41.328443 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.311389    5958 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15377h33m31.258449216s"
Oct 30 19:42:41.328443 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.311643    5958 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:42:41.328443 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.312685    5958 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:42:41.328854 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.315252    5958 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:42:41.328854 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.317732    5958 log.go:25] "Validated CRI v1 image API"
Oct 30 19:42:41.328854 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.318446    5958 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:42:41.328854 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.320762    5958 fs.go:135] Filesystem UUIDs: map[163a5595-46e5-4348-8be0-1aa8714ab61b:/dev/nvme0n1p3 1dc77a52-92bf-417d-8811-05fd2f2384d5:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 19:42:41.329004 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.320777    5958 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ac9cbb47244a9352aa7d4bd1137c93bdbcc8c4991feb2789c0a17a5aef05c8f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ac9cbb47244a9352aa7d4bd1137c93bdbcc8c4991feb2789c0a17a5aef05c8f/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/187e8ebc30af6f594a6687401273619fc110adf12b8e0113e7ee57cea71c5d2f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/187e8ebc30af6f594a6687401273619fc110adf12b8e0113e7ee57cea71c5d2f/userdata/shm major:0 minor:470 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/21b5386a6dae83c293ec44372355511c77f4f6b45f376989b85773bcbc034f3c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/21b5386a6dae83c293ec44372355511c77f4f6b45f376989b85773bcbc034f3c/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/220752f888ce8a936d7ee2a4a0ec9c6b58882fb3286025b389130741ffc82a15/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/220752f888ce8a936d7ee2a4a0ec9c6b58882fb3286025b389130741ffc82a15/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3293696b27070ca5d0335ad20881602a8e9e78295151425f79e516c3d961d962/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3293696b27070ca5d0335ad20881602a8e9e78295151425f79e516c3d961d962/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/335c8f7981f2295a9fdf43c4ac98692985725068e925efbb0c5ad065f24102fa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/335c8f7981f2295a9fdf43c4ac98692985725068e925efbb0c5ad065f24102fa/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/354a9a1161c97b185d2c234e450f09223f9a182b889c82b6a76401363a432d66/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/354a9a1161c97b185d2c234e450f09223f9a182b889c82b6a76401363a432d66/userdata/shm major:0 minor:360 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/493d40a2457d2969980842246b67511aadca0feb4648ebac9bc8683bff5adc0c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/493d40a2457d2969980842246b67511aadca0feb4648ebac9bc8683bff5adc0c/userdata/shm major:0 minor:339 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/554c4b77f9d0a7a7c176dd39e730b582e2e378b60a0607e6b63b7120b2e28a4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/554c4b77f9d0a7a7c176dd39e730b582e2e378b60a0607e6b63b7120b2e28a4a/userdata/shm major:0 minor:472 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7dc893beb226092892877d0b12c7ed7d67e66b92df4f82532a9ee892247a45eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7dc893beb226092892877d0b12c7ed7d67e66b92df4f82532a9ee892247a45eb/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8698545d4a79dded51e07962a78d7124c90fe8143c26dc272d56e3feb94ed846/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8698545d4a79dded51e07962a78d7124c90fe8143c26dc272d56e3feb94ed846/userdata/shm major:0 minor:265 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8bdd2a1db11610307f5842206c66764c9cc590f10218b9fa345ffb3c8fffd446/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8bdd2a1db11610307f5842206c66764c9cc590f10218b9fa345ffb3c8fffd446/userdata/shm major:0 minor:475 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/96ea2175c027fdae0036df0774cf8de40dc933e8677e3cf7af9de89078144182/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/96ea2175c027fdae0036df0774cf8de40dc933e8677e3cf7af9de89078144182/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e995309e7a18f11cf075a79bdc9a12bd1c7e2fb0b42bd91980500d0be2c43c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e995309e7a18f11cf075a79bdc9a12bd1c7e2fb0b42bd91980500d0be2c43c2/userdata/shm major:0 minor:345 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5c4f6e7e4882cdb016ccfefe987c7727973340cee27ecb103b4cbad9ee2ffdc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5c4f6e7e4882cdb016ccfefe987c7727973340cee27ecb103b4cbad9ee2ffdc/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/addca54b0cfe92a90da331ab9549795785599e21e8aa7599b5e8c4bccaf399e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/addca54b0cfe92a90da331ab9549795785599e21e8aa7599b5e8c4bccaf399e5/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b31aeda5cb15fecf77cf84227228897b19991c47a8ee13fdea3b6aa2bcde28a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b31aeda5cb15fecf77cf84227228897b19991c47a8ee13fdea3b6aa2bcde28a7/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b95c2d037efc75b7bec069c16ffe43df706d099dfab64de8e67b4fdd1e0f04dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b95c2d037efc75b7bec069c16ffe43df706d099dfab64de8e67b4fdd1e0f04dc/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb7ae1edbb9e66dd1e6c25b9bbca1f35d9f0749e81bbc071838660ae3cf02ce0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb7ae1edbb9e66dd1e6c25b9bbca1f35d9f0749e81bbc071838660ae3cf02ce0/userdata/shm major:0 minor:477 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca771da4e671d62f4d3e26f0a503559a4c02b532afec77ed8b75626b74cbe8ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca771da4e671d62f4d3e26f0a503559a4c02b532afec77ed8b75626b74cbe8ea/userdata/shm major:0 minor:307 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3af04fe42901cdc0b7120ac70d0486cdfdc32e2ecde57614459b61e6cbd3890/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3af04fe42901cdc0b7120ac70d0486cdfdc32e2ecde57614459b61e6cbd3890/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f309b7066d4709e2196330d74d66737ff20a52fc1f3bc1500bcaf68b20c4d8fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f309b7066d4709e2196330d74d66737ff20a52fc1f3bc1500bcaf68b20c4d8fd/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7a0c42e78195e1214da15d460376329d87d668b2a0e8568d6fd317c439b54c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7a0c42e78195e1214da15d460376329d87d668b2a0e8568d6fd317c439b54c8/userdata/shm major:0 minor:259 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f89e87a0f9d755e9e49d255b09cb96a73ca80735b93a6d55ef1db19bb737ff6b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f89e87a0f9d755e9e49d255b09cb96a73ca80735b93a6d55ef1db19bb737ff6b/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa8572cc41c0e22b72a4e7b7b0683e571485522d6f14b5913769a8fd574a2beb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa8572cc41c0e22b72a4e7b7b0683e571485522d6f14b5913769a8fd574a2beb/userdata/shm major:0 minor:332 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fda962d53dc8abc96dd0958cd718190bb56fbd1046f067a4cbb3d964bdbf1ddc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fda962d53dc8abc96dd0958cd718190bb56fbd1046f067a4cbb3d964bdbf1ddc/userdata/shm major:0 minor:353 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/023e1dd6-ffb9-48c8-a407-be580b23bddb/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/023e1dd6-ffb9-48c8-a407-be580b23bddb/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/023e1dd6-ffb9-48c8-a407-be580b23bddb/volumes/kubernetes.io~projected/kube-api-access-td8bh:{mountpoint:/var/lib/kubelet/pods/023e1dd6-ffb9-48c8-a407-be580b23bddb/volumes/kubernetes.io~projected/kube-api-access-td8bh major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/023e1dd6-ffb9-48c8-a407-be580b23bddb/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/023e1dd6-ffb9-48c8-a407-be580b23bddb/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/043ccaed-b00b-4406-b156-eb95ea8e05f3/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/043ccaed-b00b-4406-b156-eb95ea8e05f3/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/043ccaed-b00b-4406-b156-eb95ea8e05f3/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/043ccaed-b00b-4406-b156-eb95ea8e05f3/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/043ccaed-b00b-4406-b156-eb95ea8e05f3/volumes/kubernetes.io~projected/kube-api-access-g28pf:{mountpoint:/var/lib/kubelet/pods/043ccaed-b00b-4406-b156-eb95ea8e05f3/volumes/kubernetes.io~projected/kube-api-access-g28pf major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08a234af-5c01-4901-80ae-2ec70fa08bfe/volumes/kubernetes.io~projected/kube-api-access-g5ngc:{mountpoint:/var/lib/kubelet/pods/08a234af-5c01-4901-80ae-2ec70fa08bfe/volumes/kubernetes.io~projected/kube-api-access-g5ngc major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b/volumes/kubernetes.io~projected/kube-api-access-c5mbb:{mountpoint:/var/lib/kubelet/pods/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b/volumes/kubernetes.io~projected/kube-api-access-c5mbb major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f31378b-b459-4ff0-9e72-d0d4266ab7b6/volumes/kubernetes.io~projected/kube-api-access-hklwg:{mountpoint:/var/lib/kubelet/pods/0f31378b-b459-4ff0-9e72-d0d4266ab7b6/volumes/kubernetes.io~projected/kube-api-access-hklwg major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f31378b-b459-4ff0-9e72-d0d4266ab7b6/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/0f31378b-b459-4ff0-9e72-d0d4266ab7b6/volumes/kubernetes.io~secret/metrics-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14e5d4c8-c53b-43b7-8301-804226ba1867/volumes/kubernetes.io~projected/kube-api-access-v2tsl:{mountpoint:/var/lib/kubelet/pods/14e5d4c8-c53b-43b7-8301-804226ba1867/volumes/kubernetes.io~projected/kube-api-access-v2tsl major:0 minor:474 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/154facaf-4163-41fb-a6d9-fb04a5d8e12e/volumes/kubernetes.io~projected/kube-api-access-85gjj:{mountpoint:/var/lib/kubelet/pods/154facaf-4163-41fb-a6d9-fb04a5d8e12e/volumes/kubernetes.io~projected/kube-api-access-85gjj major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/154facaf-4163-41fb-a6d9-fb04a5d8e12e/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/154facaf-4163-41fb-a6d9-fb04a5d8e12e/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/154facaf-4163-41fb-a6d9-fb04a5d8e12e/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/154facaf-4163-41fb-a6d9-fb04a5d8e12e/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/154facaf-4163-41fb-a6d9-fb04a5d8e12e/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/154facaf-4163-41fb-a6d9-fb04a5d8e12e/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c4d99d2-8ba6-410c-8337-f527e98d4651/volumes/kubernetes.io~projected/kube-api-access-rc48c:{mountpoint:/var/lib/kubelet/pods/1c4d99d2-8ba6-410c-8337-f527e98d4651/volumes/kubernetes.io~projected/kube-api-access-rc48c major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21e03037-74dd-44b2-bac7-791d30cf240d/volumes/kubernetes.io~projected/kube-api-access-hk87g:{mountpoint:/var/lib/kubelet/pods/21e03037-74dd-44b2-bac7-791d30cf240d/volumes/kubernetes.io~projected/kube-api-access-hk87g major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25a55956-200b-4ca6-ad5e-050caf7116e4/volumes/kubernetes.io~projected/kube-api-access-nbcq7:{mountpoint:/var/lib/kubelet/pods/25a55956-200b-4ca6-ad5e-050caf7116e4/volumes/kubernetes.io~projected/kube-api-access-nbcq7 major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25a55956-200b-4ca6-ad5e-050caf7116e4/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/25a55956-200b-4ca6-ad5e-050caf7116e4/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25a55956-200b-4ca6-ad5e-050caf7116e4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/25a55956-200b-4ca6-ad5e-050caf7116e4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25a55956-200b-4ca6-ad5e-050caf7116e4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/25a55956-200b-4ca6-ad5e-050caf7116e4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25a55956-200b-4ca6-ad5e-050caf7116e4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/25a55956-200b-4ca6-ad5e-050caf7116e4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25a55956-200b-4ca6-ad5e-050caf7116e4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/25a55956-200b-4ca6-ad5e-050caf7116e4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25a55956-200b-4ca6-ad5e-050caf7116e4/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/25a55956-200b-4ca6-ad5e-050caf7116e4/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/419b608e-bfa8-4d2b-bacd-08c286fb7707/volumes/kubernetes.io~projected/kube-api-access-4j7rs:{mountpoint:/var/lib/kubelet/pods/419b608e-bfa8-4d2b-bacd-08c286fb7707/volumes/kubernetes.io~projected/kube-api-access-4j7rs major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/419b608e-bfa8-4d2b-bacd-08c286fb7707/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/419b608e-bfa8-4d2b-bacd-08c286fb7707/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/419b608e-bfa8-4d2b-bacd-08c286fb7707/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/419b608e-bfa8-4d2b-bacd-08c286fb7707/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56bbeb6c-dce1-42dc-9291-615c39153a7d/volumes/kubernetes.io~projected/kube-api-access-hhn8m:{mountpoint:/var/lib/kubelet/pods/56bbeb6c-dce1-42dc-9291-615c39153a7d/volumes/kubernetes.io~projected/kube-api-access-hhn8m major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56bbeb6c-dce1-42dc-9291-615c39153a7d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/56bbeb6c-dce1-42dc-9291-615c39153a7d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56bbeb6c-dce1-42dc-9291-615c39153a7d/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/56bbeb6c-dce1-42dc-9291-615c39153a7d/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5afcafb9-baea-4a5f-906a-1ac08f1f6834/volumes/kubernetes.io~projected/kube-api-access-qzjng:{mountpoint:/var/lib/kubelet/pods/5afcafb9-baea-4a5f-906a-1ac08f1f6834/volumes/kubernetes.io~projected/kube-api-access-qzjng major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5afcafb9-baea-4a5f-906a-1ac08f1f6834/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/5afcafb9-baea-4a5f-906a-1ac08f1f6834/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6391e2a6-bb94-4270-9fa9-f6d35f3e433c/volumes/kubernetes.io~projected/kube-api-access-49wqq:{mountpoint:/var/lib/kubelet/pods/6391e2a6-bb94-4270-9fa9-f6d35f3e433c/volumes/kubernetes.io~projected/kube-api-access-49wqq major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f405174-11f7-414c-a940-2a63c8989f3a/volumes/kubernetes.io~projected/kube-api-access-dgm8w:{mountpoint:/var/lib/kubelet/pods/6f405174-11f7-414c-a940-2a63c8989f3a/volumes/kubernetes.io~projected/kube-api-access-dgm8w major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f405174-11f7-414c-a940-2a63c8989f3a/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/6f405174-11f7-414c-a940-2a63c8989f3a/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7cf2adf5-da5b-4d63-a490-9e88f3598288/volumes/kubernetes.io~projected/kube-api-access-s45zp:{mountpoint:/var/lib/kubelet/pods/7cf2adf5-da5b-4d63-a490-9e88f3598288/volumes/kubernetes.io~projected/kube-api-access-s45zp major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91ef5bd9-9316-418a-a989-8557a38e0058/volumes/kubernetes.io~projected/kube-api-access-7n9wq:{mountpoint:/var/lib/kubelet/pods/91ef5bd9-9316-418a-a989-8557a38e0058/volumes/kubernetes.io~projected/kube-api-access-7n9wq major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91ef5bd9-9316-418a-a989-8557a38e0058/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/91ef5bd9-9316-418a-a989-8557a38e0058/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91ef5bd9-9316-418a-a989-8557a38e0058/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/91ef5bd9-9316-418a-a989-8557a38e0058/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91ef5bd9-9316-418a-a989-8557a38e0058/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/91ef5bd9-9316-418a-a989-8557a38e0058/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91ef5bd9-9316-418a-a989-8557a38e0058/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/91ef5bd9-9316-418a-a989-8557a38e0058/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93cfc69d-646b-4910-972d-04c9076c6c3c/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/93cfc69d-646b-4910-972d-04c9076c6c3c/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:457 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b9838bc-8604-40f5-99d5-70c240eb439d/volumes/kubernetes.io~projected/kube-api-access-cdwx2:{mountpoint:/var/lib/kubelet/pods/9b9838bc-8604-40f5-99d5-70c240eb439d/volumes/kubernetes.io~projected/kube-api-access-cdwx2 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cd48d14-cd97-4894-a43e-fa678a1833b3/volumes/kubernetes.io~projected/kube-api-access-g4xj4:{mountpoint:/var/lib/kubelet/pods/9cd48d14-cd97-4894-a43e-fa678a1833b3/volumes/kubernetes.io~projected/kube-api-access-g4xj4 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cd48d14-cd97-4894-a43e-fa678a1833b3/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/9cd48d14-cd97-4894-a43e-fa678a1833b3/volumes/kubernetes.io~secret/metrics-certs major:0 minor:453 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a94cd52c-5a75-45c0-9669-587b56e8fc70/volumes/kubernetes.io~projected/kube-api-access-k4drl:{mountpoint:/var/lib/kubelet/pods/a94cd52c-5a75-45c0-9669-587b56e8fc70/volumes/kubernetes.io~projected/kube-api-access-k4drl major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a94cd52c-5a75-45c0-9669-587b56e8fc70/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a94cd52c-5a75-45c0-9669-587b56e8fc70/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a94cd52c-5a75-45c0-9669-587b56e8fc70/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/a94cd52c-5a75-45c0-9669-587b56e8fc70/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba9ed4cb-13c2-4ead-b549-5469e2724f6c/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/ba9ed4cb-13c2-4ead-b549-5469e2724f6c/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volume-subpaths/web-config/config-reloader/2 major:0 minor:466 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:466 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volume-subpaths/web-config/prometheus/15 major:0 minor:466 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~empty-dir/config-out major:0 minor:462 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~projected/kube-api-access-5kp8z:{mountpoint:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~projected/kube-api-access-5kp8z major:0 minor:469 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~projected/tls-assets major:0 minor:458 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/config major:0 minor:461 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:460 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:464 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:463 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:468 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:467 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:459 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:465 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/web-config major:0 minor:466 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volume-subpaths/web-config/alertmanager/9 major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volume-subpaths/web-config/config-reloader/6 major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~empty-dir/config-out major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~projected/kube-api-access-v8b99:{mountpoint:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~projected/kube-api-access-v8b99 major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~projected/tls-assets major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~secret/config-volume major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~secret/web-config major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7a30ded-2282-462e-bd34-dd4ad735a3ca/volumes/kubernetes.io~projected/kube-api-access-cktlc:{mountpoint:/var/lib/kubelet/pods/f7a30ded-2282-462e-bd34-dd4ad735a3ca/volumes/kubernetes.io~projected/kube-api-access-cktlc major:0 minor:56 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/74110d48425bb2f642ba3b766591a0dd828d084e7e8ada12dbe073809bcf7a57/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/000e74319789f22184e8f1c790ed824e34000ad909f4b79817baca5d30cfa0be/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/2034cbb8bf4430cc25b156c75405d909f705b863d65b5de00b304de1a96e5a80/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/0123bdba8fd79f4d505e3851e181ad41a9525b583e34db410e82ff084f7bc8fb/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/3cd9e23be89c7bd5ae75ce626e7aab67d9290fb859676008b7b769358956c78e/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-112:{mountpoint:/var/lib/containers/storage/overlay/7649822dbac1fc4e9a03ea11b3d378dec9af4e05cd9d7783a4734d96663e50e8/merged major:0 minor:112 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/e9857dcbf2bffadb93433737d1d05636efe3fe2249c34fae3fb843a716c6e214/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/f4faaa771e8a8c615b1d559f1f25478a42b5ed888dea718d868176d852346376/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/0df58b9167adec230efbc36ed03b5dc42e04c522143486ce13d3d071d2775539/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/c1eb528055b39834f7661dce96b267278e416d036dd776e8be3d64e1af806144/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/654afd7ce7c42e42efc2f84568bb4316d34afdff3f79bbcef86a0c835a3811d6/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/e0559d7f607e17313dd13b4587a81d49d2a00e8183e795fc46b675b5a379f7ce/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/78a01233c9ea888e141e6ed6e33feff29e41140efbb275cc2eaa5c95270ad8d8/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/b1cbc45efe63048ca402eee19763d6d43f7681270b11f69dd050022e006746de/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/fbbf6bb90363861093defb900e550bf29a3ba178b33ea3310dab3ab86f396e4f/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/99e897ee41d56bc09255841e419bb4ea9de7980ec5ef3b27724fff54cd2bbc66/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/3c4c14d418c17ba81e4783c11eaa6adba994d24dbb5d653e99a19594350e06ca/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/398dad915418de8518fd682e87171e473a0a504aed262d7dbea83fea43da91b8/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/824e197639d3e135f0cb1a84953d8f36caf00b4481ed1d5a8e4d649a29f5eb53/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c87e4d2fa65c53cec1c8d0cdaaa9d167a538ca6efb2b1969f30db2c3712dba17/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/956331e1a6352de6badfe1ae8fee29437e8f18380337cbb42fac0976a0018a30/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/332b91165a818052b84f984c1d59f1a1fce711988667489e791ec9d4e7ea277b/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/84af01018b95aadfb50473745821b93e24cab16e506fdd065e250de5c81300c6/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/9a64e273c051b73b4b05a43ad693b11645148c506a4adad367bf97418272eee1/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/f4aa03c1aa8b9d50963184b68c307fe32586e9b1be41b36c56072305919edb36/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/ff723a3b4a191da14e032f6568fa08f52f031bfb6cc1d518350a36be42e72e7a/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/4f845014264c50720d62a2048b87fb47a26f14515380a32a7130ce263079abbb/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/bfabedc6142f038ee34567a2d1fc93de244a5eae8c1102f5741ba172f2b0e69d/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/d7346b9410c653d1b37ff804ae66f8156948c00311cc878469c9dd6f923a24aa/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/f1179e5e04a2ffa440ee3d2a767f55e563187f960de27fb840be14a6f61794d8/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/70583d4907f5e53e2ad64f174bad2f34eb46fa05cb28fb075dce2db75db03014/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/ec8aa763060e5afb6da5330df8bb96b0b45d42c1d0952a03a8ecdb15784d7b53/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/2bb6af4acda28a335897da5015fb1125c22d68cb1b516afaef24706e458b51aa/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/6eea6a5dab7a73761f50fa68cf4b7b408b05d8c7fb73ff883f46476ecccc3d76/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/c04984df7837150a0777731bf36cca14805dd1f8b6fb881a01d8006233f6751c/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/8c126f8267bc0f63ec05bba1645418d531129400004d7b52a8dd25b110207590/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/d03cb192fcd8fa124d9318eafacf9c30dc0fc8e0760154cf7225974edab23baa/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/e6e0b5f20b248255c528c2e26079433d2c2ca046d8805adc874b086bcc9428a4/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/3a20ff880b69f899b38235aedb876057213a5df81b283ece718a49d5af2d8d40/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/186329ca85d16e7dbb2588d53108d24c6e90e905009794b0583113072a397293/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/e18962cade029ab465094b7084e46d3564dbb4af8691169c20cd6d92fe75bfa2/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/724163297eae0fc318764d28cb942d154052360f02d0fffa78ce439393400233/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/82ec81b92df1f9472c6fc5867b156fb2493abc7ab67ffcdb161cda076e690db1/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/922567c942cd9dae1f6ce5c1d8269d4e762c833fb8c3ee098b5e32b6bb8adc2f/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/166359acc036da645a3edf5ee26b81e76b6f1963dad355ff9c059d0e6ad7c290/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/3927f0848dd7145273ac2322413c34eb0acce039051a9a6fccc4e604a084f045/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/9f7dc2ff1c97b2db90106881f69188aec1601c65d9773ad4fa8762d024c96881/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/b139d56f62902b4b917a82e03caf7c1350f0b7cbd0c985c63936d351916ac20c/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/8a62ab5efb11f7d8f43a4964eba8200f0039c9a2e8a572c77fa0c39773dc7125/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/1666759b5e1fa7b3c174a267c2d714a5ac8a3372afcdebc13b3c6589ac8b9a45/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/31c8330008da1750914dd577dd36fce37475a67d99d4cda36fb6563974ac17b8/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-384:{mountpoint:/var/lib/containers/storage/overlay/eda7ed03324f5c0c1374c3c427320aa1bc1e5ef5e5262df664a2d3d6d1703f10/merged major:0 minor:384 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/8ccac4e84edad5ffd3a9168ef2cd6545ee2713a5e66554afcc1046fc2f1f774e/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-395:{mountpoint:/var/lib/containers/storage/overlay/2cc8ca0c3d337add3c377810dc6cc509515120ae6777dd2ddf64a52d1a446d26/merged major:0 minor:395 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/cbd32cb9c0279b1540cf4de6136f040b8e3d07ab648d04604b782ce7ff9c870d/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/5834f54eca9fc4666527dc843362b031b78eacd086ecefaf040a7fb978f98aa6/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/40da64e759d949e081987107e96bce86bc9e5f704efa19c15d95461dc955802b/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/d892212fff8c1dca13d7ab780bcc236f75513daea3b664bfc52131a56b79d1d2/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/3d19f2f25cea7f6967b098b3c6478f071ee1dabf6f1293e457c96b30ed5a9ad0/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/fde7930e74aef696ba16255de7ce643b87b459967ee01f98df83b2ae462bf7bf/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/6597bf09f415b22873ba735adc10020e696e8b7372e0893785a4a0dd3e1f497e/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-437:{mountpoint:/var/lib/containers/storage/overlay/b9be980290d446eba8de86428798f3de4efd7bfeb28dbc5252d6a11542a9549e/merged major:0 minor:437 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/7598ad8a9cbabffe25f0f6ed19c65d6f778f03c5a52dcd2d7275034c0b782f0e/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-448:{mountpoint:/var/lib/containers/storage/overlay/0e6a43ea1591030367551a6c6545d0bc4ef405e9470a03e30a2356aa16ffd254/merged major:0 minor:448 fsType:overlay blockSize:0} overlay_0-479:{mountpoint:/var/lib/containers/storage/overlay/d503097474f2f80de749c4b543c51e5ed8f6f0d1ec3224225af37088552a3f16/merged major:0 minor:479 fsType:overlay blockSize:0} overlay_0-481:{mountpoint:/var/lib/containers/storage/overlay/5b0fd8cee389d89784a0549c8079c886e27809725c49d89988e51dfa7056c8ff/merged major:0 minor:481 fsType:overlay blockSize:0} overlay_0-483:{mountpoint:/var/lib/containers/storage/overlay/1ab01fad12bf7917136ef13372d2d1eef07903329adaf8d8f7084744ef197d46/merged major:0 minor:483 fsType:overlay blockSize:0} overlay_0-485:{mountpoint:/var/lib/containers/storage/overlay/8c40257094e39687e40df6a15164646f9bfaf8d19807b25c5fc8719f151e4ad1/merged major:0 minor:485 fsType:overlay blockSize:0} overlay_0-487:{mountpoint:/var/lib/containers/storage/overlay/24f9217d6fb772fd73412e13d2db52eefd87bfae5aee0c10bcc50f31ca3627a9/merged major:0 minor:487 fsType:overlay blockSize:0} overlay_0-489:{mountpoint:/var/lib/containers/storage/overlay/cac08b9c2d882499d2a78c8dd992caaf1077346db95d36e2504aa138a018ca18/merged major:0 minor:489 fsType:overlay blockSize:0} overlay_0-495:{mountpoint:/var/lib/containers/storage/overlay/93e933decc249a4f02fd47ccdac0bcc34298a5f70539c437f2093f63a422bf73/merged major:0 minor:495 fsType:overlay blockSize:0} overlay_0-500:{mountpoint:/var/lib/containers/storage/overlay/c91d84d2c5debc02104d02e524817351edeecc6ad2c86fb1a2328a0594eb57df/merged major:0 minor:500 fsType:overlay blockSize:0} overlay_0-506:{mountpoint:/var/lib/containers/storage/overlay/687b50e50001bcc1cedcd2310884684c6d0ae1cecb632d8e112c9b0d3a1996ba/merged major:0 minor:506 fsType:overlay blockSize:0} overlay_0-512:{mountpoint:/var/lib/containers/storage/overlay/27ffb78897347a011f4c7ca0a542d980534e66c82b42bce782185458b5ea3c4d/merged major:0 minor:512 fsType:overlay blockSize:0} overlay_0-517:{mountpoint:/var/lib/containers/storage/overlay/c1f09d7f241757bfd3c70629787dbb135ebd737994ec1a5f19b2784b43f91fb9/merged major:0 minor:517 fsType:overlay blockSize:0} overlay_0-522:{mountpoint:/var/lib/containers/storage/overlay/0001176f5224462f2f0a2193cf617350e82f88de3491f1d5c48ae92ffc336af9/merged major:0 minor:522 fsType:overlay blockSize:0} overlay_0-527:{mountpoint:/var/lib/containers/storage/overlay/3c017cdcf5ac7b739881e7b8db09c4f80758bf9e2a0b897b268191adb22d9c1f/merged major:0 minor:527 fsType:overlay blockSize:0} overlay_0-532:{mountpoint:/var/lib/containers/storage/overlay/a752546c655952194b2b61db9330a265c32917df6e97bff21447d374bfc435e9/merged major:0 minor:532 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/972ba095e05ee1580dac2c0c0a5d5de5920d7004b8b6540395acc9f01db35af8/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/53b7733874dce6e4d39a976adffc4ad4448f65ad2377418f5995737cd38c3490/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/a706b7afa94c428c3715744aa9db1717cb1bc43188bab12f4657c2b106b4866f/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/460dd8e447dd03d5c02e4e4f98ecf4bef0e9a18f3d1911bfaa1296c37aac01d6/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/c3c0f398558f1a58487dc689aaf7c0c51d841f571795d7d41ead0f18b6381423/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/2907cffe10600a242926d0ec7c0b4df7e93ba3ba2a599205bd0e7de67900d8e4/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/908f1fdc26bc9d14f585183a22abb6dd4d2864b269b33292befb0b9a51b9f6c4/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/cc153ebdc9bcfc256c3a29c6d9282cb569ce88ed7b20e4629ae81ff17ab1371a/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/8c33a595bf1e56b4355b5eb4d11782753d716f8224f91e9a51b8e3525bed7bcd/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/b7dacad6bf5942f256761d117e18b80192b04798a2858b2a621fbab5742b05d8/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 19:42:41.329391 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.328944    5958 manager.go:217] Machine: {Timestamp:2026-10-30 19:42:41.328461772 +0000 UTC m=+0.060677166 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200284 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f1a0197ff298c62c0f3881f4e119c SystemUUID:ec2f1a01-97ff-298c-62c0-f3881f4e119c BootID:64fdb448-a4e4-4170-a207-4dee29803448 Filesystems:[{Device:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:305 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:303 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/91ef5bd9-9316-418a-a989-8557a38e0058/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:356 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:466 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f89e87a0f9d755e9e49d255b09cb96a73ca80735b93a6d55ef1db19bb737ff6b/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:299 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:460 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ac9cbb47244a9352aa7d4bd1137c93bdbcc8c4991feb2789c0a17a5aef05c8f/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25a55956-200b-4ca6-ad5e-050caf7116e4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:327 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:468 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b31aeda5cb15fecf77cf84227228897b19991c47a8ee13fdea3b6aa2bcde28a7/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/25a55956-200b-4ca6-ad5e-050caf7116e4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:329 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:458 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:466 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb7ae1edbb9e66dd1e6c25b9bbca1f35d9f0749e81bbc071838660ae3cf02ce0/userdata/shm DeviceMajor:0 DeviceMinor:477 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/043ccaed-b00b-4406-b156-eb95ea8e05f3/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/023e1dd6-ffb9-48c8-a407-be580b23bddb/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:305 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/25a55956-200b-4ca6-ad5e-050caf7116e4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:330 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3af04fe42901cdc0b7120ac70d0486cdfdc32e2ecde57614459b61e6cbd3890/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:463 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/addca54b0cfe92a90da331ab9549795785599e21e8aa7599b5e8c4bccaf399e5/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:462 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-483 DeviceMajor:0 DeviceMinor:483 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-500 DeviceMajor:0 DeviceMinor:500 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:466 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9cd48d14-cd97-4894-a43e-fa678a1833b3/volumes/kubernetes.io~projected/kube-api-access-g4xj4 DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6f405174-11f7-414c-a940-2a63c8989f3a/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:343 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e995309e7a18f11cf075a79bdc9a12bd1c7e2fb0b42bd91980500d0be2c43c2/userdata/shm DeviceMajor:0 DeviceMinor:345 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-384 DeviceMajor:0 DeviceMinor:384 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7a30ded-2282-462e-bd34-dd4ad735a3ca/volumes/kubernetes.io~projected/kube-api-access-cktlc DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a94cd52c-5a75-45c0-9669-587b56e8fc70/volumes/kubernetes.io~projected/kube-api-access-k4drl DeviceMajor:0 DeviceMinor:252 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8698545d4a79dded51e07962a78d7124c90fe8143c26dc272d56e3feb94ed846/userdata/shm DeviceMajor:0 DeviceMinor:265 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25a55956-200b-4ca6-ad5e-050caf7116e4/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:326 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0f31378b-b459-4ff0-9e72-d0d4266ab7b6/volumes/kubernetes.io~projected/kube-api-access-hklwg DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/56bbeb6c-dce1-42dc-9291-615c39153a7d/volumes/kubernetes.io~projected/kube-api-access-hhn8m DeviceMajor:0 DeviceMinor:258 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14e5d4c8-c53b-43b7-8301-804226ba1867/volumes/kubernetes.io~projected/kube-api-access-v2tsl DeviceMajor:0 DeviceMinor:474 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-506 DeviceMajor:0 DeviceMinor:506 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7dc893beb226092892877d0b12c7ed7d67e66b92df4f82532a9ee892247a45eb/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25a55956-200b-4ca6-ad5e-050caf7116e4/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:331 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:299 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/043ccaed-b00b-4406-b156-eb95ea8e05f3/volumes/kubernetes.io~projected/kube-api-access-g28pf DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f309b7066d4709e2196330d74d66737ff20a52fc1f3bc1500bcaf68b20c4d8fd/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/220752f888ce8a936d7ee2a4a0ec9c6b58882fb3286025b389130741ffc82a15/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-112 DeviceMajor:0 DeviceMinor:112 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/419b608e-bfa8-4d2b-bacd-08c286fb7707/volumes/kubernetes.io~projected/kube-api-access-4j7rs DeviceMajor:0 DeviceMinor:257 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7cf2adf5-da5b-4d63-a490-9e88f3598288/volumes/kubernetes.io~projected/kube-api-access-s45zp DeviceMajor:0 DeviceMinor:338 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a94cd52c-5a75-45c0-9669-587b56e8fc70/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:247 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6f405174-11f7-414c-a940-2a63c8989f3a/volumes/kubernetes.io~projected/kube-api-access-dgm8w DeviceMajor:0 DeviceMinor:344 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/154facaf-4163-41fb-a6d9-fb04a5d8e12e/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:348 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/154facaf-4163-41fb-a6d9-fb04a5d8e12e/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:349 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/93cfc69d-646b-4910-972d-04c9076c6c3c/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:457 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8bdd2a1db11610307f5842206c66764c9cc590f10218b9fa345ffb3c8fffd446/userdata/shm DeviceMajor:0 DeviceMinor:475 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-495 DeviceMajor:0 DeviceMinor:495 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/419b608e-bfa8-4d2b-bacd-08c286fb7707/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:255 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:299 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:461 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5afcafb9-baea-4a5f-906a-1ac08f1f6834/volumes/kubernetes.io~projected/kube-api-access-qzjng DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/023e1dd6-ffb9-48c8-a407-be580b23bddb/volumes/kubernetes.io~projected/kube-api-access-td8bh DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6391e2a6-bb94-4270-9fa9-f6d35f3e433c/volumes/kubernetes.io~projected/kube-api-access-49wqq DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/023e1dd6-ffb9-48c8-a407-be580b23bddb/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/5afcafb9-baea-4a5f-906a-1ac08f1f6834/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:301 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:294 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:302 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9cd48d14-cd97-4894-a43e-fa678a1833b3/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:453 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~projected/kube-api-access-5kp8z DeviceMajor:0 DeviceMinor:469 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-527 DeviceMajor:0 DeviceMinor:527 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5c4f6e7e4882cdb016ccfefe987c7727973340cee27ecb103b4cbad9ee2ffdc/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/91ef5bd9-9316-418a-a989-8557a38e0058/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:355 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:467 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-485 DeviceMajor:0 DeviceMinor:485 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:299 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-522 DeviceMajor:0 DeviceMinor:522 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a94cd52c-5a75-45c0-9669-587b56e8fc70/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:251 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-448 DeviceMajor:0 DeviceMinor:448 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21e03037-74dd-44b2-bac7-791d30cf240d/volumes/kubernetes.io~projected/kube-api-access-hk87g DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3293696b27070ca5d0335ad20881602a8e9e78295151425f79e516c3d961d962/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/21b5386a6dae83c293ec44372355511c77f4f6b45f376989b85773bcbc034f3c/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/91ef5bd9-9316-418a-a989-8557a38e0058/volumes/kubernetes.io~projected/kube-api-access-7n9wq DeviceMajor:0 DeviceMinor:359 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/354a9a1161c97b185d2c234e450f09223f9a182b889c82b6a76401363a432d66/userdata/shm DeviceMajor:0 DeviceMinor:360 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-479 DeviceMajor:0 DeviceMinor:479 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-481 DeviceMajor:0 DeviceMinor:481 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/08a234af-5c01-4901-80ae-2ec70fa08bfe/volumes/kubernetes.io~projected/kube-api-access-g5ngc DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1c4d99d2-8ba6-410c-8337-f527e98d4651/volumes/kubernetes.io~projected/kube-api-access-rc48c DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/493d40a2457d2969980842246b67511aadca0feb4648ebac9bc8683bff5adc0c/userdata/shm DeviceMajor:0 DeviceMinor:339 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/154facaf-4163-41fb-a6d9-fb04a5d8e12e/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:347 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-395 DeviceMajor:0 DeviceMinor:395 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/ba9ed4cb-13c2-4ead-b549-5469e2724f6c/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/56bbeb6c-dce1-42dc-9291-615c39153a7d/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:253 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7a0c42e78195e1214da15d460376329d87d668b2a0e8568d6fd317c439b54c8/userdata/shm DeviceMajor:0 DeviceMinor:259 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:459 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b95c2d037efc75b7bec069c16ffe43df706d099dfab64de8e67b4fdd1e0f04dc/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/96ea2175c027fdae0036df0774cf8de40dc933e8677e3cf7af9de89078144182/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~projected/kube-api-access-v8b99 DeviceMajor:0 DeviceMinor:306 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/91ef5bd9-9316-418a-a989-8557a38e0058/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:357 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:464 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-487 DeviceMajor:0 DeviceMinor:487 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b/volumes/kubernetes.io~projected/kube-api-access-c5mbb DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0f31378b-b459-4ff0-9e72-d0d4266ab7b6/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca771da4e671d62f4d3e26f0a503559a4c02b532afec77ed8b75626b74cbe8ea/userdata/shm DeviceMajor:0 DeviceMinor:307 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/25a55956-200b-4ca6-ad5e-050caf7116e4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:328 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:465 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/043ccaed-b00b-4406-b156-eb95ea8e05f3/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa8572cc41c0e22b72a4e7b7b0683e571485522d6f14b5913769a8fd574a2beb/userdata/shm DeviceMajor:0 DeviceMinor:332 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/91ef5bd9-9316-418a-a989-8557a38e0058/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:358 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/187e8ebc30af6f594a6687401273619fc110adf12b8e0113e7ee57cea71c5d2f/userdata/shm DeviceMajor:0 DeviceMinor:470 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/554c4b77f9d0a7a7c176dd39e730b582e2e378b60a0607e6b63b7120b2e28a4a/userdata/shm DeviceMajor:0 DeviceMinor:472 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-489 DeviceMajor:0 DeviceMinor:489 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b9838bc-8604-40f5-99d5-70c240eb439d/volumes/kubernetes.io~projected/kube-api-access-cdwx2 DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/335c8f7981f2295a9fdf43c4ac98692985725068e925efbb0c5ad065f24102fa/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:304 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/56bbeb6c-dce1-42dc-9291-615c39153a7d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:254 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:466 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-517 DeviceMajor:0 DeviceMinor:517 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/419b608e-bfa8-4d2b-bacd-08c286fb7707/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:256 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-437 DeviceMajor:0 DeviceMinor:437 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-512 DeviceMajor:0 DeviceMinor:512 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-532 DeviceMajor:0 DeviceMinor:532 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:298 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:300 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/25a55956-200b-4ca6-ad5e-050caf7116e4/volumes/kubernetes.io~projected/kube-api-access-nbcq7 DeviceMajor:0 DeviceMinor:322 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/154facaf-4163-41fb-a6d9-fb04a5d8e12e/volumes/kubernetes.io~projected/kube-api-access-85gjj DeviceMajor:0 DeviceMinor:350 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fda962d53dc8abc96dd0958cd718190bb56fbd1046f067a4cbb3d964bdbf1ddc/userdata/shm DeviceMajor:0 DeviceMinor:353 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0ac9cbb47244a93 MacAddress:42:28:a2:4d:0f:e1 Speed:10000 Mtu:8901} {Name:187e8ebc30af6f5 MacAddress:9a:dd:49:d1:52:09 Speed:10000 Mtu:8901} {Name:21b5386a6dae83c MacAddress:ae:69:c2:a2:25:ec Speed:10000 Mtu:8901} {Name:3293696b27070ca MacAddress:7e:fe:7b:ab:cd:73 Speed:10000 Mtu:8901} {Name:354a9a1161c97b1 MacAddress:aa:31:ba:23:d7:ac Speed:10000 Mtu:8901} {Name:493d40a2457d296 MacAddress:ba:8c:69:da:ce:68 Speed:10000 Mtu:8901} {Name:554c4b77f9d0a7a MacAddress:ce:98:8d:81:b7:02 Speed:10000 Mtu:8901} {Name:8698545d4a79dde MacAddress:0e:7e:1a:63:4f:4b Speed:10000 Mtu:8901} {Name:8bdd2a1db116103 MacAddress:72:12:fb:d3:8b:55 Speed:10000 Mtu:8901} {Name:9e995309e7a18f1 MacAddress:8a:9c:ed:52:42:67 Speed:10000 Mtu:8901} {Name:addca54b0cfe92a MacAddress:36:34:bb:3c:62:ab Speed:10000 Mtu:8901} {Name:bb7ae1edbb9e66d MacAddress:3a:74:58:fb:5a:3a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:31:2a:33:2a:8b Speed:0 Mtu:9001} {Name:br-int MacAddress:da:82:97:a9:df:49 Speed:0 Mtu:8901} {Name:ca771da4e671d62 MacAddress:f2:87:8c:56:b2:4c Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:31:2a:33:2a:8b Speed:0 Mtu:9001} {Name:fa8572cc41c0e22 MacAddress:06:ac:fe:50:88:cd Speed:10000 Mtu:8901} {Name:fda962d53dc8abc MacAddress:e2:6a:0b:7b:89:3f Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:22:f4:5e:f7:28:7b Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ee:f8:ab:cf:4c:58 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:42:41.329685 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.329394    5958 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:42:41.329685 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.329440    5958 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:42:41.329775 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.329752    5958 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:42:41.329909 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.329775    5958 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-240.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:42:41.329950 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.329916    5958 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:42:41.329950 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.329924    5958 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:42:41.329950 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.329939    5958 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:42:41.330025 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.329953    5958 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:42:41.330064 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.330044    5958 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:42:41.330228 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.330218    5958 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:42:41.330341 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.330333    5958 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:42:41.330376 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.330369    5958 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:42:41.330404 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.330384    5958 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:42:41.330404 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.330393    5958 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:42:41.330404 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.330402    5958 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:42:41.330515 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.330411    5958 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:42:41.332696 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.332216    5958 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:42:41.332696 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.332245    5958 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:42:41.333066 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.333047    5958 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:42:41.346987 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.346960    5958 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:42:41.346987 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.346980    5958 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:42:41.347156 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.347091    5958 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 19:42:41.347304 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.347291    5958 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:42:41.347342 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.347321    5958 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:42:41.347441 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.347431    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:42:41.347497 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.347446    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:42:41.347497 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.347466    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:42:41.347497 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.347472    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:42:41.347497 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.347477    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:42:41.347497 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.347483    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:42:41.347497 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.347488    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:42:41.347497 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.347494    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:42:41.347729 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.347500    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:42:41.347729 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.347506    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:42:41.347729 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.347520    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:42:41.347729 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.347531    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:42:41.347729 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.347554    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:42:41.347729 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.347561    5958 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:42:41.351014 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.350997    5958 server.go:1263] "Started kubelet"
Oct 30 19:42:41.351135 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.351105    5958 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:42:41.351188 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.351099    5958 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:42:41.351188 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.351181    5958 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:42:41.351554 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.351527    5958 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:42:41.351592 ip-10-0-142-240 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:42:41.352526 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.352509    5958 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:42:41.356750 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.356725    5958 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:42:41.356859 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.356808    5958 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:42:41.356940 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.356901    5958 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:36:29 +0000 UTC" deadline="2028-04-02 04:00:00.283253206 +0000 UTC"
Oct 30 19:42:41.356940 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.356930    5958 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12464h17m18.926326834s"
Oct 30 19:42:41.357061 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.356972    5958 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:42:41.357061 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.356986    5958 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:42:41.357061 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.357034    5958 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:42:41.357423 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:42:41.357390    5958 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-240.us-west-1.compute.internal\" not found"
Oct 30 19:42:41.361424 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.361388    5958 factory.go:55] Registering systemd factory
Oct 30 19:42:41.361424 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.361413    5958 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.361901    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="419b608e-bfa8-4d2b-bacd-08c286fb7707" volumeName="kubernetes.io/empty-dir/419b608e-bfa8-4d2b-bacd-08c286fb7707-volume-directive-shadow" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.361956    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f405174-11f7-414c-a940-2a63c8989f3a" volumeName="kubernetes.io/secret/6f405174-11f7-414c-a940-2a63c8989f3a-monitoring-plugin-cert" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.361977    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91ef5bd9-9316-418a-a989-8557a38e0058" volumeName="kubernetes.io/configmap/91ef5bd9-9316-418a-a989-8557a38e0058-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.361990    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a94cd52c-5a75-45c0-9669-587b56e8fc70" volumeName="kubernetes.io/configmap/a94cd52c-5a75-45c0-9669-587b56e8fc70-metrics-client-ca" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362006    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="043ccaed-b00b-4406-b156-eb95ea8e05f3" volumeName="kubernetes.io/projected/043ccaed-b00b-4406-b156-eb95ea8e05f3-kube-api-access-g28pf" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362024    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56bbeb6c-dce1-42dc-9291-615c39153a7d" volumeName="kubernetes.io/secret/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362036    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" volumeName="kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-web-config" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362055    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d79cb0a2-da98-4c41-a201-50755411d3c1" volumeName="kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362074    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14e5d4c8-c53b-43b7-8301-804226ba1867" volumeName="kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362092    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cf2adf5-da5b-4d63-a490-9e88f3598288" volumeName="kubernetes.io/projected/7cf2adf5-da5b-4d63-a490-9e88f3598288-kube-api-access-s45zp" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362109    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a94cd52c-5a75-45c0-9669-587b56e8fc70" volumeName="kubernetes.io/secret/a94cd52c-5a75-45c0-9669-587b56e8fc70-openshift-state-metrics-tls" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362122    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="419b608e-bfa8-4d2b-bacd-08c286fb7707" volumeName="kubernetes.io/configmap/419b608e-bfa8-4d2b-bacd-08c286fb7707-metrics-client-ca" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362140    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21e03037-74dd-44b2-bac7-791d30cf240d" volumeName="kubernetes.io/projected/21e03037-74dd-44b2-bac7-791d30cf240d-kube-api-access-hk87g" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362154    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b81aa0b-b68a-4225-81e6-0e5d59d5c63b" volumeName="kubernetes.io/secret/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362212    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b9838bc-8604-40f5-99d5-70c240eb439d" volumeName="kubernetes.io/projected/9b9838bc-8604-40f5-99d5-70c240eb439d-kube-api-access-cdwx2" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362230    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" volumeName="kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362249    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d79cb0a2-da98-4c41-a201-50755411d3c1" volumeName="kubernetes.io/empty-dir/d79cb0a2-da98-4c41-a201-50755411d3c1-config-out" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362264    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f31378b-b459-4ff0-9e72-d0d4266ab7b6" volumeName="kubernetes.io/configmap/0f31378b-b459-4ff0-9e72-d0d4266ab7b6-config-volume" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362278    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c4d99d2-8ba6-410c-8337-f527e98d4651" volumeName="kubernetes.io/projected/1c4d99d2-8ba6-410c-8337-f527e98d4651-kube-api-access-rc48c" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362296    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" volumeName="kubernetes.io/projected/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-tls-assets" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362309    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" volumeName="kubernetes.io/projected/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-kube-api-access-5kp8z" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362325    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d79cb0a2-da98-4c41-a201-50755411d3c1" volumeName="kubernetes.io/empty-dir/d79cb0a2-da98-4c41-a201-50755411d3c1-alertmanager-main-db" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362338    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5afcafb9-baea-4a5f-906a-1ac08f1f6834" volumeName="kubernetes.io/projected/5afcafb9-baea-4a5f-906a-1ac08f1f6834-kube-api-access-qzjng" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362352    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6391e2a6-bb94-4270-9fa9-f6d35f3e433c" volumeName="kubernetes.io/configmap/6391e2a6-bb94-4270-9fa9-f6d35f3e433c-serviceca" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362371    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="154facaf-4163-41fb-a6d9-fb04a5d8e12e" volumeName="kubernetes.io/secret/154facaf-4163-41fb-a6d9-fb04a5d8e12e-secret-metrics-server-tls" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362384    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a55956-200b-4ca6-ad5e-050caf7116e4" volumeName="kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362401    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="419b608e-bfa8-4d2b-bacd-08c286fb7707" volumeName="kubernetes.io/configmap/419b608e-bfa8-4d2b-bacd-08c286fb7707-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362413    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91ef5bd9-9316-418a-a989-8557a38e0058" volumeName="kubernetes.io/projected/91ef5bd9-9316-418a-a989-8557a38e0058-kube-api-access-7n9wq" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362441    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cd48d14-cd97-4894-a43e-fa678a1833b3" volumeName="kubernetes.io/projected/9cd48d14-cd97-4894-a43e-fa678a1833b3-kube-api-access-g4xj4" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362474    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" volumeName="kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362510    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b81aa0b-b68a-4225-81e6-0e5d59d5c63b" volumeName="kubernetes.io/configmap/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362527    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f31378b-b459-4ff0-9e72-d0d4266ab7b6" volumeName="kubernetes.io/empty-dir/0f31378b-b459-4ff0-9e72-d0d4266ab7b6-tmp-dir" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362551    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56bbeb6c-dce1-42dc-9291-615c39153a7d" volumeName="kubernetes.io/configmap/56bbeb6c-dce1-42dc-9291-615c39153a7d-metrics-client-ca" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362565    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cd48d14-cd97-4894-a43e-fa678a1833b3" volumeName="kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362578    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" volumeName="kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-config" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362597    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d79cb0a2-da98-4c41-a201-50755411d3c1" volumeName="kubernetes.io/projected/d79cb0a2-da98-4c41-a201-50755411d3c1-kube-api-access-v8b99" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362612    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a55956-200b-4ca6-ad5e-050caf7116e4" volumeName="kubernetes.io/configmap/25a55956-200b-4ca6-ad5e-050caf7116e4-metrics-client-ca" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362631    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a55956-200b-4ca6-ad5e-050caf7116e4" volumeName="kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-tls" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362644    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" volumeName="kubernetes.io/empty-dir/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-db" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362657    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56bbeb6c-dce1-42dc-9291-615c39153a7d" volumeName="kubernetes.io/projected/56bbeb6c-dce1-42dc-9291-615c39153a7d-kube-api-access-hhn8m" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362677    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c4d99d2-8ba6-410c-8337-f527e98d4651" volumeName="kubernetes.io/configmap/1c4d99d2-8ba6-410c-8337-f527e98d4651-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362691    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="043ccaed-b00b-4406-b156-eb95ea8e05f3" volumeName="kubernetes.io/empty-dir/043ccaed-b00b-4406-b156-eb95ea8e05f3-tmp" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362708    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f31378b-b459-4ff0-9e72-d0d4266ab7b6" volumeName="kubernetes.io/projected/0f31378b-b459-4ff0-9e72-d0d4266ab7b6-kube-api-access-hklwg" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362721    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7a30ded-2282-462e-bd34-dd4ad735a3ca" volumeName="kubernetes.io/configmap/f7a30ded-2282-462e-bd34-dd4ad735a3ca-iptables-alerter-script" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362735    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="023e1dd6-ffb9-48c8-a407-be580b23bddb" volumeName="kubernetes.io/projected/023e1dd6-ffb9-48c8-a407-be580b23bddb-kube-api-access-td8bh" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362757    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a55956-200b-4ca6-ad5e-050caf7116e4" volumeName="kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-grpc-tls" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362771    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" volumeName="kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-tls" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362788    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a55956-200b-4ca6-ad5e-050caf7116e4" volumeName="kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362804    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91ef5bd9-9316-418a-a989-8557a38e0058" volumeName="kubernetes.io/secret/91ef5bd9-9316-418a-a989-8557a38e0058-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362818    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="154facaf-4163-41fb-a6d9-fb04a5d8e12e" volumeName="kubernetes.io/empty-dir/154facaf-4163-41fb-a6d9-fb04a5d8e12e-audit-log" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362833    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a55956-200b-4ca6-ad5e-050caf7116e4" volumeName="kubernetes.io/projected/25a55956-200b-4ca6-ad5e-050caf7116e4-kube-api-access-nbcq7" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362844    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56bbeb6c-dce1-42dc-9291-615c39153a7d" volumeName="kubernetes.io/configmap/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362860    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" volumeName="kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362872    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d79cb0a2-da98-4c41-a201-50755411d3c1" volumeName="kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-cluster-tls-config" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362883    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f405174-11f7-414c-a940-2a63c8989f3a" volumeName="kubernetes.io/projected/6f405174-11f7-414c-a940-2a63c8989f3a-kube-api-access-dgm8w" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362899    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91ef5bd9-9316-418a-a989-8557a38e0058" volumeName="kubernetes.io/configmap/91ef5bd9-9316-418a-a989-8557a38e0058-telemeter-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362928    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d79cb0a2-da98-4c41-a201-50755411d3c1" volumeName="kubernetes.io/configmap/d79cb0a2-da98-4c41-a201-50755411d3c1-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362945    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d79cb0a2-da98-4c41-a201-50755411d3c1" volumeName="kubernetes.io/projected/d79cb0a2-da98-4c41-a201-50755411d3c1-tls-assets" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362959    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c4d99d2-8ba6-410c-8337-f527e98d4651" volumeName="kubernetes.io/configmap/1c4d99d2-8ba6-410c-8337-f527e98d4651-multus-daemon-config" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362969    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="419b608e-bfa8-4d2b-bacd-08c286fb7707" volumeName="kubernetes.io/projected/419b608e-bfa8-4d2b-bacd-08c286fb7707-kube-api-access-4j7rs" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362984    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" volumeName="kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.362995    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93cfc69d-646b-4910-972d-04c9076c6c3c" volumeName="kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363012    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a94cd52c-5a75-45c0-9669-587b56e8fc70" volumeName="kubernetes.io/secret/a94cd52c-5a75-45c0-9669-587b56e8fc70-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363023    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="154facaf-4163-41fb-a6d9-fb04a5d8e12e" volumeName="kubernetes.io/configmap/154facaf-4163-41fb-a6d9-fb04a5d8e12e-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363036    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="154facaf-4163-41fb-a6d9-fb04a5d8e12e" volumeName="kubernetes.io/projected/154facaf-4163-41fb-a6d9-fb04a5d8e12e-kube-api-access-85gjj" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363057    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91ef5bd9-9316-418a-a989-8557a38e0058" volumeName="kubernetes.io/secret/91ef5bd9-9316-418a-a989-8557a38e0058-secret-telemeter-client" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363071    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="023e1dd6-ffb9-48c8-a407-be580b23bddb" volumeName="kubernetes.io/secret/023e1dd6-ffb9-48c8-a407-be580b23bddb-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363089    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="154facaf-4163-41fb-a6d9-fb04a5d8e12e" volumeName="kubernetes.io/secret/154facaf-4163-41fb-a6d9-fb04a5d8e12e-secret-metrics-server-client-certs" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363101    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="419b608e-bfa8-4d2b-bacd-08c286fb7707" volumeName="kubernetes.io/secret/419b608e-bfa8-4d2b-bacd-08c286fb7707-kube-state-metrics-tls" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363113    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91ef5bd9-9316-418a-a989-8557a38e0058" volumeName="kubernetes.io/configmap/91ef5bd9-9316-418a-a989-8557a38e0058-metrics-client-ca" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363130    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d79cb0a2-da98-4c41-a201-50755411d3c1" volumeName="kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363156    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7cf2adf5-da5b-4d63-a490-9e88f3598288" volumeName="kubernetes.io/empty-dir/7cf2adf5-da5b-4d63-a490-9e88f3598288-tmp" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363176    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" volumeName="kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363188    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d79cb0a2-da98-4c41-a201-50755411d3c1" volumeName="kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363202    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" volumeName="kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363223    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="023e1dd6-ffb9-48c8-a407-be580b23bddb" volumeName="kubernetes.io/configmap/023e1dd6-ffb9-48c8-a407-be580b23bddb-ovnkube-config" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363237    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="154facaf-4163-41fb-a6d9-fb04a5d8e12e" volumeName="kubernetes.io/secret/154facaf-4163-41fb-a6d9-fb04a5d8e12e-client-ca-bundle" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363259    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6391e2a6-bb94-4270-9fa9-f6d35f3e433c" volumeName="kubernetes.io/projected/6391e2a6-bb94-4270-9fa9-f6d35f3e433c-kube-api-access-49wqq" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363271    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91ef5bd9-9316-418a-a989-8557a38e0058" volumeName="kubernetes.io/secret/91ef5bd9-9316-418a-a989-8557a38e0058-federate-client-tls" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363285    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba9ed4cb-13c2-4ead-b549-5469e2724f6c" volumeName="kubernetes.io/secret/ba9ed4cb-13c2-4ead-b549-5469e2724f6c-agent-certs" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363304    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" volumeName="kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363318    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" volumeName="kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-metrics-client-certs" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363345    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a55956-200b-4ca6-ad5e-050caf7116e4" volumeName="kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363359    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d79cb0a2-da98-4c41-a201-50755411d3c1" volumeName="kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-web-config" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363372    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5afcafb9-baea-4a5f-906a-1ac08f1f6834" volumeName="kubernetes.io/secret/5afcafb9-baea-4a5f-906a-1ac08f1f6834-cert" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363391    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08a234af-5c01-4901-80ae-2ec70fa08bfe" volumeName="kubernetes.io/projected/08a234af-5c01-4901-80ae-2ec70fa08bfe-kube-api-access-g5ngc" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363403    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b81aa0b-b68a-4225-81e6-0e5d59d5c63b" volumeName="kubernetes.io/projected/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-kube-api-access-c5mbb" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363419    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21e03037-74dd-44b2-bac7-791d30cf240d" volumeName="kubernetes.io/configmap/21e03037-74dd-44b2-bac7-791d30cf240d-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363432    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a55956-200b-4ca6-ad5e-050caf7116e4" volumeName="kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363444    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56bbeb6c-dce1-42dc-9291-615c39153a7d" volumeName="kubernetes.io/secret/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-tls" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363481    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7a30ded-2282-462e-bd34-dd4ad735a3ca" volumeName="kubernetes.io/projected/f7a30ded-2282-462e-bd34-dd4ad735a3ca-kube-api-access-cktlc" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363493    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="419b608e-bfa8-4d2b-bacd-08c286fb7707" volumeName="kubernetes.io/secret/419b608e-bfa8-4d2b-bacd-08c286fb7707-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363547    5958 factory.go:153] Registering CRI-O factory
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363563    5958 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363623    5958 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363642    5958 factory.go:103] Registering Raw factory
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363658    5958 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:42:41.364052 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.363988    5958 manager.go:319] Starting recovery of all containers
Oct 30 19:42:41.367971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.364612    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21e03037-74dd-44b2-bac7-791d30cf240d" volumeName="kubernetes.io/configmap/21e03037-74dd-44b2-bac7-791d30cf240d-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 19:42:41.367971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.364648    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a55956-200b-4ca6-ad5e-050caf7116e4" volumeName="kubernetes.io/empty-dir/25a55956-200b-4ca6-ad5e-050caf7116e4-tmp" seLinuxMountContext=""
Oct 30 19:42:41.367971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.364661    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" volumeName="kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 19:42:41.367971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.364674    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56bbeb6c-dce1-42dc-9291-615c39153a7d" volumeName="kubernetes.io/empty-dir/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-textfile" seLinuxMountContext=""
Oct 30 19:42:41.367971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.364686    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b9838bc-8604-40f5-99d5-70c240eb439d" volumeName="kubernetes.io/empty-dir/9b9838bc-8604-40f5-99d5-70c240eb439d-tmp-dir" seLinuxMountContext=""
Oct 30 19:42:41.367971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.364697    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba9ed4cb-13c2-4ead-b549-5469e2724f6c" volumeName="kubernetes.io/configmap/ba9ed4cb-13c2-4ead-b549-5469e2724f6c-konnectivity-ca" seLinuxMountContext=""
Oct 30 19:42:41.367971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.364708    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" volumeName="kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Oct 30 19:42:41.367971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.364719    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" volumeName="kubernetes.io/empty-dir/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-config-out" seLinuxMountContext=""
Oct 30 19:42:41.367971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.364732    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" volumeName="kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-grpc-tls" seLinuxMountContext=""
Oct 30 19:42:41.367971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.364759    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" volumeName="kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-thanos-prometheus-http-client-file" seLinuxMountContext=""
Oct 30 19:42:41.367971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.364779    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="043ccaed-b00b-4406-b156-eb95ea8e05f3" volumeName="kubernetes.io/empty-dir/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-tuned" seLinuxMountContext=""
Oct 30 19:42:41.367971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.364790    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b81aa0b-b68a-4225-81e6-0e5d59d5c63b" volumeName="kubernetes.io/empty-dir/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-data-volume" seLinuxMountContext=""
Oct 30 19:42:41.367971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.364801    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21e03037-74dd-44b2-bac7-791d30cf240d" volumeName="kubernetes.io/configmap/21e03037-74dd-44b2-bac7-791d30cf240d-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 19:42:41.367971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.364813    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" volumeName="kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-metrics-client-ca" seLinuxMountContext=""
Oct 30 19:42:41.367971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.364824    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d79cb0a2-da98-4c41-a201-50755411d3c1" volumeName="kubernetes.io/configmap/d79cb0a2-da98-4c41-a201-50755411d3c1-metrics-client-ca" seLinuxMountContext=""
Oct 30 19:42:41.367971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.364835    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a94cd52c-5a75-45c0-9669-587b56e8fc70" volumeName="kubernetes.io/projected/a94cd52c-5a75-45c0-9669-587b56e8fc70-kube-api-access-k4drl" seLinuxMountContext=""
Oct 30 19:42:41.367971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.364848    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d79cb0a2-da98-4c41-a201-50755411d3c1" volumeName="kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-config-volume" seLinuxMountContext=""
Oct 30 19:42:41.367971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.364858    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d79cb0a2-da98-4c41-a201-50755411d3c1" volumeName="kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-main-tls" seLinuxMountContext=""
Oct 30 19:42:41.367971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.364869    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="023e1dd6-ffb9-48c8-a407-be580b23bddb" volumeName="kubernetes.io/configmap/023e1dd6-ffb9-48c8-a407-be580b23bddb-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 19:42:41.367971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.364881    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="023e1dd6-ffb9-48c8-a407-be580b23bddb" volumeName="kubernetes.io/configmap/023e1dd6-ffb9-48c8-a407-be580b23bddb-env-overrides" seLinuxMountContext=""
Oct 30 19:42:41.367971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.364917    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="154facaf-4163-41fb-a6d9-fb04a5d8e12e" volumeName="kubernetes.io/configmap/154facaf-4163-41fb-a6d9-fb04a5d8e12e-metrics-server-audit-profiles" seLinuxMountContext=""
Oct 30 19:42:41.367971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.364933    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91ef5bd9-9316-418a-a989-8557a38e0058" volumeName="kubernetes.io/secret/91ef5bd9-9316-418a-a989-8557a38e0058-telemeter-client-tls" seLinuxMountContext=""
Oct 30 19:42:41.367971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.364950    5958 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f31378b-b459-4ff0-9e72-d0d4266ab7b6" volumeName="kubernetes.io/secret/0f31378b-b459-4ff0-9e72-d0d4266ab7b6-metrics-tls" seLinuxMountContext=""
Oct 30 19:42:41.367971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.364962    5958 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:42:41.367971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.364971    5958 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:42:41.388812 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.388753    5958 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:42:41.390484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.390436    5958 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:42:41.390484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.390483    5958 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:42:41.390682 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.390506    5958 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:42:41.390682 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:42:41.390543    5958 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 19:42:41.395424 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.395395    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 19:42:41.398139 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.398116    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 19:42:41.404185 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.404157    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 19:42:41.404593 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.404567    5958 generic.go:356] "Generic (PLEG): container finished" podID="023e1dd6-ffb9-48c8-a407-be580b23bddb" containerID="d4033ab8d6dea9638d45016197f050f6c2301cba6416845eaf1b8cefc629562c" exitCode=1
Oct 30 19:42:41.410252 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.410215    5958 generic.go:356] "Generic (PLEG): container finished" podID="ce4e2caa0da71bc2cbc3d09da16fb366" containerID="83694aedc05138d37a3b8060b490230c635d02b7bfe840e27b996a719aa83bc5" exitCode=0
Oct 30 19:42:41.424271 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.424240    5958 generic.go:356] "Generic (PLEG): container finished" podID="21e03037-74dd-44b2-bac7-791d30cf240d" containerID="2dff488a7e1fa75df4990bb69934e7de07ef114c525dac2113b18e4dab359061" exitCode=0
Oct 30 19:42:41.424271 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.424265    5958 generic.go:356] "Generic (PLEG): container finished" podID="21e03037-74dd-44b2-bac7-791d30cf240d" containerID="b88fdc18114d418ea6aaaf0a9767d0dcc137368b14a6725d9213a635bd3681a8" exitCode=0
Oct 30 19:42:41.424271 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.424274    5958 generic.go:356] "Generic (PLEG): container finished" podID="21e03037-74dd-44b2-bac7-791d30cf240d" containerID="7cce0678bfa61b6a91d3486d915c1ca32b7ad56b201c48a5c4770c50aabe582d" exitCode=0
Oct 30 19:42:41.424492 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.424279    5958 generic.go:356] "Generic (PLEG): container finished" podID="21e03037-74dd-44b2-bac7-791d30cf240d" containerID="17ebe86da4b369a2ef173f3eecebd573696dc1812fe83cc0c52a81b3258a8b27" exitCode=0
Oct 30 19:42:41.424492 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.424285    5958 generic.go:356] "Generic (PLEG): container finished" podID="21e03037-74dd-44b2-bac7-791d30cf240d" containerID="20c327eb7f04b6b3110fb9250fbfeca9bf224d39e5001d8b0dc758db20881812" exitCode=0
Oct 30 19:42:41.424492 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.424290    5958 generic.go:356] "Generic (PLEG): container finished" podID="21e03037-74dd-44b2-bac7-791d30cf240d" containerID="4e422d7ace23a35ebcfb19012524b52e0842c53ff40bfc8b57fced0bfea0c845" exitCode=0
Oct 30 19:42:41.436970 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.436933    5958 generic.go:356] "Generic (PLEG): container finished" podID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerID="a9af2f1df48ee00d2017099c3228cbb04f43cef68b474005219ce36c44e86c10" exitCode=0
Oct 30 19:42:41.443972 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.443935    5958 generic.go:356] "Generic (PLEG): container finished" podID="56bbeb6c-dce1-42dc-9291-615c39153a7d" containerID="f4042ad646245e91d971a4165882870ec5260ee6f3f43f6fb10dc07470202097" exitCode=0
Oct 30 19:42:41.449544 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.449468    5958 generic.go:356] "Generic (PLEG): container finished" podID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerID="83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef" exitCode=0
Oct 30 19:42:41.457489 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:42:41.457463    5958 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-240.us-west-1.compute.internal\" not found"
Oct 30 19:42:41.490606 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.490576    5958 manager.go:324] Recovery completed
Oct 30 19:42:41.490769 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:42:41.490630    5958 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 19:42:41.515386 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.515359    5958 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:42:41.517568 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.517548    5958 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:42:41.517626 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.517583    5958 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:42:41.517626 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.517593    5958 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:42:41.520786 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.520760    5958 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:42:41.520786 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.520781    5958 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:42:41.520968 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.520782    5958 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:42:41.520968 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.520846    5958 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:42:41.521045 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.521030    5958 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 19:42:41.521074 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.521046    5958 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 19:42:41.521074 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.521064    5958 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 19:42:41.521074 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.521070    5958 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 19:42:41.521074 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.521075    5958 policy_none.go:50] "Start"
Oct 30 19:42:41.521176 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.521086    5958 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:42:41.521176 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.521096    5958 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:42:41.521252 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.521242    5958 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 19:42:41.521280 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.521253    5958 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 19:42:41.521280 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.521258    5958 policy_none.go:44] "Start"
Oct 30 19:42:41.521795 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.521778    5958 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:42:41.521871 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.521812    5958 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:42:41.521871 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.521826    5958 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:42:41.526389 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.526356    5958 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:42:41.530208 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.530193    5958 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:42:41.530283 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:42:41.530226    5958 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:42:41.530283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.530240    5958 server.go:87] "Starting device plugin registration server"
Oct 30 19:42:41.530531 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.530519    5958 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:42:41.530580 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.530532    5958 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:42:41.530673 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.530636    5958 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:42:41.530809 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.530776    5958 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:42:41.530876 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.530826    5958 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:42:41.530876 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.530834    5958 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:42:41.531528 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.530890    5958 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:42:41.530808127Z","UUID":"a36f3f6f-e21c-42ad-b949-ce28b5074000","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:42:41.532953 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.532903    5958 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:42:41.533059 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.532986    5958 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:42:41.533555 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:42:41.533510    5958 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:42:41.533641 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:42:41.533579    5958 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-240.us-west-1.compute.internal\" not found"
Oct 30 19:42:41.557799 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.557761    5958 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:42:41.558075 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.558058    5958 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 19:42:41.603084 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.603050    5958 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:42:41.608138 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.608107    5958 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:42:41.631386 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.631353    5958 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:42:41.632510 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.632490    5958 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:42:41.632585 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.632523    5958 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:42:41.632585 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.632532    5958 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-240.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:42:41.632671 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.632607    5958 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:42:41.652997 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.652969    5958 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:42:41.690800 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.690737    5958 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal"]
Oct 30 19:42:41.691431 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.691391    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal" event={"ID":"84c50bb941905481e1c090101140d6a9","Type":"ContainerStarted","Data":"779c78b1607fc2a35a721852f8b8545ca3c80556f73164d1c93a1271bed817c8"}
Oct 30 19:42:41.691500 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.691442    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal" event={"ID":"84c50bb941905481e1c090101140d6a9","Type":"ContainerStarted","Data":"f309b7066d4709e2196330d74d66737ff20a52fc1f3bc1500bcaf68b20c4d8fd"}
Oct 30 19:42:41.691500 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.691486    5958 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:42:41.691576 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.691490    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal" event={"ID":"ce4e2caa0da71bc2cbc3d09da16fb366","Type":"ContainerStarted","Data":"b33c8243ee6e5bdff8f7299ba6c0d27065dc647181ddc5e63f1040702d34f069"}
Oct 30 19:42:41.691619 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.691591    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal" event={"ID":"ce4e2caa0da71bc2cbc3d09da16fb366","Type":"ContainerDied","Data":"83694aedc05138d37a3b8060b490230c635d02b7bfe840e27b996a719aa83bc5"}
Oct 30 19:42:41.691619 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.691612    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal" event={"ID":"ce4e2caa0da71bc2cbc3d09da16fb366","Type":"ContainerStarted","Data":"335c8f7981f2295a9fdf43c4ac98692985725068e925efbb0c5ad065f24102fa"}
Oct 30 19:42:41.691686 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.691655    5958 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:42:41.753958 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.753887    5958 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:42:41.753958 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.753958    5958 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:42:41.756717 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.756697    5958 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:42:41.756798 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:42:41.756744    5958 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:42:41.758869 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.758848    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ce4e2caa0da71bc2cbc3d09da16fb366-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal\" (UID: \"ce4e2caa0da71bc2cbc3d09da16fb366\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:42:41.758919 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.758876    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce4e2caa0da71bc2cbc3d09da16fb366-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal\" (UID: \"ce4e2caa0da71bc2cbc3d09da16fb366\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:42:41.758919 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.758894    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84c50bb941905481e1c090101140d6a9-config\") pod \"kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal\" (UID: \"84c50bb941905481e1c090101140d6a9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:42:41.759523 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.759507    5958 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:42:41.759564 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:42:41.759547    5958 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:42:41.859178 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.859137    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84c50bb941905481e1c090101140d6a9-config\") pod \"kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal\" (UID: \"84c50bb941905481e1c090101140d6a9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:42:41.859178 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.859178    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ce4e2caa0da71bc2cbc3d09da16fb366-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal\" (UID: \"ce4e2caa0da71bc2cbc3d09da16fb366\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:42:41.859383 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.859201    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce4e2caa0da71bc2cbc3d09da16fb366-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal\" (UID: \"ce4e2caa0da71bc2cbc3d09da16fb366\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:42:41.859383 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.859252    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce4e2caa0da71bc2cbc3d09da16fb366-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal\" (UID: \"ce4e2caa0da71bc2cbc3d09da16fb366\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:42:41.859383 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.859254    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84c50bb941905481e1c090101140d6a9-config\") pod \"kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal\" (UID: \"84c50bb941905481e1c090101140d6a9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:42:41.859383 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:41.859254    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ce4e2caa0da71bc2cbc3d09da16fb366-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal\" (UID: \"ce4e2caa0da71bc2cbc3d09da16fb366\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:42:42.331095 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.331056    5958 apiserver.go:52] "Watching apiserver"
Oct 30 19:42:42.435828 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.435785    5958 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:42:42.436564 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.436537    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-snlg9","openshift-network-operator/iptables-alerter-dxw48","openshift-cluster-node-tuning-operator/tuned-qz2rj","openshift-monitoring/telemeter-client-596848ccc9-tsm2b","openshift-network-diagnostics/network-check-target-m9t6w","openshift-ingress-canary/ingress-canary-b7jpb","openshift-ovn-kubernetes/ovnkube-node-ft6pg","openshift-dns/node-resolver-5qzfj","openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl","kube-system/konnectivity-agent-rwbsr","openshift-monitoring/monitoring-plugin-77f4f76594-p527g","openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl","openshift-multus/multus-l2mtk","openshift-monitoring/prometheus-k8s-0","openshift-dns/dns-default-rjdlc","openshift-insights/insights-runtime-extractor-t4b2n","kube-system/kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal","openshift-monitoring/alertmanager-main-0","kube-system/global-pull-secret-syncer-8t48c","openshift-monitoring/metrics-server-8698dc55b4-pwqbt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-rj77q","openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97","openshift-monitoring/node-exporter-t586q","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52","openshift-console/downloads-6dd789c8b7-zk9rp","openshift-image-registry/node-ca-9x594"]
Oct 30 19:42:42.459857 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.459826    5958 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:42:42.462810 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.462787    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-run-openvswitch\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.462902 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.462828    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f31378b-b459-4ff0-9e72-d0d4266ab7b6-config-volume\") pod \"dns-default-rjdlc\" (UID: \"0f31378b-b459-4ff0-9e72-d0d4266ab7b6\") " pod="openshift-dns/dns-default-rjdlc"
Oct 30 19:42:42.462902 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.462853    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d79cb0a2-da98-4c41-a201-50755411d3c1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.463027 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.462929    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-sysconfig\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.463027 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.462950    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-run\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.463027 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.462997    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-cnibin\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.463027 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463025    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c4d99d2-8ba6-410c-8337-f527e98d4651-cni-binary-copy\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.463190 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463088    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk87g\" (UniqueName: \"kubernetes.io/projected/21e03037-74dd-44b2-bac7-791d30cf240d-kube-api-access-hk87g\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:42:42.463190 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463115    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/56bbeb6c-dce1-42dc-9291-615c39153a7d-root\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:42.463190 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463140    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdwx2\" (UniqueName: \"kubernetes.io/projected/9b9838bc-8604-40f5-99d5-70c240eb439d-kube-api-access-cdwx2\") pod \"node-resolver-5qzfj\" (UID: \"9b9838bc-8604-40f5-99d5-70c240eb439d\") " pod="openshift-dns/node-resolver-5qzfj"
Oct 30 19:42:42.463291 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463184    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5afcafb9-baea-4a5f-906a-1ac08f1f6834-cert\") pod \"ingress-canary-b7jpb\" (UID: \"5afcafb9-baea-4a5f-906a-1ac08f1f6834\") " pod="openshift-ingress-canary/ingress-canary-b7jpb"
Oct 30 19:42:42.463291 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463263    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-run-multus-certs\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.463291 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463287    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-etc-selinux\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:42:42.463404 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463306    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5ngc\" (UniqueName: \"kubernetes.io/projected/08a234af-5c01-4901-80ae-2ec70fa08bfe-kube-api-access-g5ngc\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:42:42.463437 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463415    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhn8m\" (UniqueName: \"kubernetes.io/projected/56bbeb6c-dce1-42dc-9291-615c39153a7d-kube-api-access-hhn8m\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:42.463491 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463441    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.463491 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463485    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/419b608e-bfa8-4d2b-bacd-08c286fb7707-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:42.463568 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463502    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-textfile\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:42.463568 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463522    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cf2adf5-da5b-4d63-a490-9e88f3598288-tmp\") pod \"downloads-6dd789c8b7-zk9rp\" (UID: \"7cf2adf5-da5b-4d63-a490-9e88f3598288\") " pod="openshift-console/downloads-6dd789c8b7-zk9rp"
Oct 30 19:42:42.463646 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463582    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21e03037-74dd-44b2-bac7-791d30cf240d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:42:42.463646 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463601    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/419b608e-bfa8-4d2b-bacd-08c286fb7707-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:42.463646 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463608    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-textfile\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:42.463646 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463608    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cf2adf5-da5b-4d63-a490-9e88f3598288-tmp\") pod \"downloads-6dd789c8b7-zk9rp\" (UID: \"7cf2adf5-da5b-4d63-a490-9e88f3598288\") " pod="openshift-console/downloads-6dd789c8b7-zk9rp"
Oct 30 19:42:42.463646 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463623    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5mbb\" (UniqueName: \"kubernetes.io/projected/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-kube-api-access-c5mbb\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:42.463830 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463653    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d79cb0a2-da98-4c41-a201-50755411d3c1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.463830 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463712    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9b9838bc-8604-40f5-99d5-70c240eb439d-hosts-file\") pod \"node-resolver-5qzfj\" (UID: \"9b9838bc-8604-40f5-99d5-70c240eb439d\") " pod="openshift-dns/node-resolver-5qzfj"
Oct 30 19:42:42.463830 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463740    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j7rs\" (UniqueName: \"kubernetes.io/projected/419b608e-bfa8-4d2b-bacd-08c286fb7707-kube-api-access-4j7rs\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:42.463830 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463771    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-tls\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:42.463830 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463814    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91ef5bd9-9316-418a-a989-8557a38e0058-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:42.464049 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463833    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:42:42.464049 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463857    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/56bbeb6c-dce1-42dc-9291-615c39153a7d-metrics-client-ca\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:42.464049 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463891    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49wqq\" (UniqueName: \"kubernetes.io/projected/6391e2a6-bb94-4270-9fa9-f6d35f3e433c-kube-api-access-49wqq\") pod \"node-ca-9x594\" (UID: \"6391e2a6-bb94-4270-9fa9-f6d35f3e433c\") " pod="openshift-image-registry/node-ca-9x594"
Oct 30 19:42:42.464049 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463922    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/21e03037-74dd-44b2-bac7-791d30cf240d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:42:42.464049 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463940    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/154facaf-4163-41fb-a6d9-fb04a5d8e12e-audit-log\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:42.464049 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463957    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-sysctl-conf\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.464049 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.463973    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-multus-cni-dir\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.464049 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464027    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/154facaf-4163-41fb-a6d9-fb04a5d8e12e-audit-log\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:42.464049 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464029    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-var-lib-kubelet\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.464506 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464065    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ba9ed4cb-13c2-4ead-b549-5469e2724f6c-konnectivity-ca\") pod \"konnectivity-agent-rwbsr\" (UID: \"ba9ed4cb-13c2-4ead-b549-5469e2724f6c\") " pod="kube-system/konnectivity-agent-rwbsr"
Oct 30 19:42:42.464506 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464080    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-config\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.464506 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464095    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.464506 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464115    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-crio-socket\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:42.464506 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464140    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-web-config\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.464506 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464162    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-etc-openvswitch\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.464506 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464197    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-run-ovn\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.464506 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464214    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-web-config\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.464506 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464229    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-run-netns\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.464506 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464263    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21e03037-74dd-44b2-bac7-791d30cf240d-cnibin\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:42:42.464506 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464299    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td8bh\" (UniqueName: \"kubernetes.io/projected/023e1dd6-ffb9-48c8-a407-be580b23bddb-kube-api-access-td8bh\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.464506 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464334    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:42.464506 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464354    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/91ef5bd9-9316-418a-a989-8557a38e0058-federate-client-tls\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:42.464506 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464371    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-socket-dir\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:42:42.464506 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464386    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d79cb0a2-da98-4c41-a201-50755411d3c1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.464506 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464401    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-data-volume\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:42.464506 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464418    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-device-dir\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:42:42.464506 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464483    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-sys\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.464506 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464506    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g28pf\" (UniqueName: \"kubernetes.io/projected/043ccaed-b00b-4406-b156-eb95ea8e05f3-kube-api-access-g28pf\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.464506 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464508    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-data-volume\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464523    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/023e1dd6-ffb9-48c8-a407-be580b23bddb-ovnkube-config\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464543    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-multus-socket-dir-parent\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464563    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-var-lib-cni-multus\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464582    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464599    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-host\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464645    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-run-systemd\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464682    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464713    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-sys-fs\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464736    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/93cfc69d-646b-4910-972d-04c9076c6c3c-kubelet-config\") pod \"global-pull-secret-syncer-8t48c\" (UID: \"93cfc69d-646b-4910-972d-04c9076c6c3c\") " pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464750    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56bbeb6c-dce1-42dc-9291-615c39153a7d-sys\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464782    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/043ccaed-b00b-4406-b156-eb95ea8e05f3-tmp\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464821    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbcq7\" (UniqueName: \"kubernetes.io/projected/25a55956-200b-4ca6-ad5e-050caf7116e4-kube-api-access-nbcq7\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464847    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s45zp\" (UniqueName: \"kubernetes.io/projected/7cf2adf5-da5b-4d63-a490-9e88f3598288-kube-api-access-s45zp\") pod \"downloads-6dd789c8b7-zk9rp\" (UID: \"7cf2adf5-da5b-4d63-a490-9e88f3598288\") " pod="openshift-console/downloads-6dd789c8b7-zk9rp"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464874    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ba9ed4cb-13c2-4ead-b549-5469e2724f6c-agent-certs\") pod \"konnectivity-agent-rwbsr\" (UID: \"ba9ed4cb-13c2-4ead-b549-5469e2724f6c\") " pod="kube-system/konnectivity-agent-rwbsr"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464889    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/043ccaed-b00b-4406-b156-eb95ea8e05f3-tmp\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464903    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464931    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-sysctl-d\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464959    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.464997    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-system-cni-dir\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.465033    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.465059    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.465116    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-tls\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.465138    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-cni-netd\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.465198    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.465218    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.465242    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgm8w\" (UniqueName: \"kubernetes.io/projected/6f405174-11f7-414c-a940-2a63c8989f3a-kube-api-access-dgm8w\") pod \"monitoring-plugin-77f4f76594-p527g\" (UID: \"6f405174-11f7-414c-a940-2a63c8989f3a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p527g"
Oct 30 19:42:42.465300 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.465278    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:42.466225 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.465394    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8b99\" (UniqueName: \"kubernetes.io/projected/d79cb0a2-da98-4c41-a201-50755411d3c1-kube-api-access-v8b99\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.466225 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.465427    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-var-lib-kubelet\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.466225 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.465471    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-kubelet\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.466225 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.465521    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:42.466225 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.465577    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25a55956-200b-4ca6-ad5e-050caf7116e4-tmp\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:42.466225 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.465606    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-multus-conf-dir\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.466225 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.465629    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/154facaf-4163-41fb-a6d9-fb04a5d8e12e-secret-metrics-server-client-certs\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:42.466225 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.465646    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f31378b-b459-4ff0-9e72-d0d4266ab7b6-metrics-tls\") pod \"dns-default-rjdlc\" (UID: \"0f31378b-b459-4ff0-9e72-d0d4266ab7b6\") " pod="openshift-dns/dns-default-rjdlc"
Oct 30 19:42:42.466225 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.465654    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25a55956-200b-4ca6-ad5e-050caf7116e4-tmp\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:42.466225 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.465667    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-wtmp\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:42.466225 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.465686    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-systemd-units\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.466225 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.465771    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-log-socket\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.466225 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.465802    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-cni-bin\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.466225 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.465839    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-var-lib-cni-bin\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.466225 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.465870    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc48c\" (UniqueName: \"kubernetes.io/projected/1c4d99d2-8ba6-410c-8337-f527e98d4651-kube-api-access-rc48c\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.466225 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.465899    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-accelerators-collector-config\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:42.466225 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.465919    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85gjj\" (UniqueName: \"kubernetes.io/projected/154facaf-4163-41fb-a6d9-fb04a5d8e12e-kube-api-access-85gjj\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:42.466891 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.466346    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-kubernetes\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.466891 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.466373    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-config-out\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.466891 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.466392    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-slash\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.466891 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.466418    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-grpc-tls\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:42.466891 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.466441    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-run-netns\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.466891 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.466482    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.466891 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.466484    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-config-out\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.466891 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.466510    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4drl\" (UniqueName: \"kubernetes.io/projected/a94cd52c-5a75-45c0-9669-587b56e8fc70-kube-api-access-k4drl\") pod \"openshift-state-metrics-7bd7489957-wd2gl\" (UID: \"a94cd52c-5a75-45c0-9669-587b56e8fc70\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl"
Oct 30 19:42:42.466891 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.466536    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cktlc\" (UniqueName: \"kubernetes.io/projected/f7a30ded-2282-462e-bd34-dd4ad735a3ca-kube-api-access-cktlc\") pod \"iptables-alerter-dxw48\" (UID: \"f7a30ded-2282-462e-bd34-dd4ad735a3ca\") " pod="openshift-network-operator/iptables-alerter-dxw48"
Oct 30 19:42:42.466891 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.466561    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.466891 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.466661    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-systemd\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.466891 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.466717    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6391e2a6-bb94-4270-9fa9-f6d35f3e433c-serviceca\") pod \"node-ca-9x594\" (UID: \"6391e2a6-bb94-4270-9fa9-f6d35f3e433c\") " pod="openshift-image-registry/node-ca-9x594"
Oct 30 19:42:42.466891 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.466769    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzjng\" (UniqueName: \"kubernetes.io/projected/5afcafb9-baea-4a5f-906a-1ac08f1f6834-kube-api-access-qzjng\") pod \"ingress-canary-b7jpb\" (UID: \"5afcafb9-baea-4a5f-906a-1ac08f1f6834\") " pod="openshift-ingress-canary/ingress-canary-b7jpb"
Oct 30 19:42:42.466891 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.466790    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-run-k8s-cni-cncf-io\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.466891 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.466806    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-etc-kubernetes\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.466891 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.466825    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91ef5bd9-9316-418a-a989-8557a38e0058-telemeter-trusted-ca-bundle\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:42.466891 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.466847    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/154facaf-4163-41fb-a6d9-fb04a5d8e12e-metrics-server-audit-profiles\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:42.466891 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.466889    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/91ef5bd9-9316-418a-a989-8557a38e0058-secret-telemeter-client\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:42.467538 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.466959    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7a30ded-2282-462e-bd34-dd4ad735a3ca-host-slash\") pod \"iptables-alerter-dxw48\" (UID: \"f7a30ded-2282-462e-bd34-dd4ad735a3ca\") " pod="openshift-network-operator/iptables-alerter-dxw48"
Oct 30 19:42:42.467538 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.466978    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-modprobe-d\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.467538 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.466995    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.467538 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467058    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c4d99d2-8ba6-410c-8337-f527e98d4651-multus-daemon-config\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.467538 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467077    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2tsl\" (UniqueName: \"kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl\") pod \"network-check-target-m9t6w\" (UID: \"14e5d4c8-c53b-43b7-8301-804226ba1867\") " pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:42:42.467538 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467131    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:42.467538 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467194    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-config-volume\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.467538 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467223    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-run-ovn-kubernetes\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.467538 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467250    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25a55956-200b-4ca6-ad5e-050caf7116e4-metrics-client-ca\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:42.467538 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467300    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.467538 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467334    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.467538 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467371    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.467538 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467401    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a94cd52c-5a75-45c0-9669-587b56e8fc70-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wd2gl\" (UID: \"a94cd52c-5a75-45c0-9669-587b56e8fc70\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl"
Oct 30 19:42:42.467538 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467433    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a94cd52c-5a75-45c0-9669-587b56e8fc70-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wd2gl\" (UID: \"a94cd52c-5a75-45c0-9669-587b56e8fc70\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl"
Oct 30 19:42:42.467538 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467486    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f31378b-b459-4ff0-9e72-d0d4266ab7b6-tmp-dir\") pod \"dns-default-rjdlc\" (UID: \"0f31378b-b459-4ff0-9e72-d0d4266ab7b6\") " pod="openshift-dns/dns-default-rjdlc"
Oct 30 19:42:42.467538 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467513    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d79cb0a2-da98-4c41-a201-50755411d3c1-config-out\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.467538 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467539    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.468181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467566    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-var-lib-openvswitch\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.468181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467554    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f31378b-b459-4ff0-9e72-d0d4266ab7b6-tmp-dir\") pod \"dns-default-rjdlc\" (UID: \"0f31378b-b459-4ff0-9e72-d0d4266ab7b6\") " pod="openshift-dns/dns-default-rjdlc"
Oct 30 19:42:42.468181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467596    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-node-log\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.468181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467622    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d79cb0a2-da98-4c41-a201-50755411d3c1-config-out\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.468181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467635    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/023e1dd6-ffb9-48c8-a407-be580b23bddb-ovn-node-metrics-cert\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.468181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467669    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6391e2a6-bb94-4270-9fa9-f6d35f3e433c-host\") pod \"node-ca-9x594\" (UID: \"6391e2a6-bb94-4270-9fa9-f6d35f3e433c\") " pod="openshift-image-registry/node-ca-9x594"
Oct 30 19:42:42.468181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467697    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/023e1dd6-ffb9-48c8-a407-be580b23bddb-env-overrides\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.468181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467739    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/419b608e-bfa8-4d2b-bacd-08c286fb7707-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:42.468181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467769    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:42.468181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467787    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-hostroot\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.468181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467804    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.468181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467849    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.468181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467878    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f7a30ded-2282-462e-bd34-dd4ad735a3ca-iptables-alerter-script\") pod \"iptables-alerter-dxw48\" (UID: \"f7a30ded-2282-462e-bd34-dd4ad735a3ca\") " pod="openshift-network-operator/iptables-alerter-dxw48"
Oct 30 19:42:42.468181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467905    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.468181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467910    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n9wq\" (UniqueName: \"kubernetes.io/projected/91ef5bd9-9316-418a-a989-8557a38e0058-kube-api-access-7n9wq\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:42.468181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467944    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs\") pod \"network-metrics-daemon-rj77q\" (UID: \"9cd48d14-cd97-4894-a43e-fa678a1833b3\") " pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:42:42.468181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.467986    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4xj4\" (UniqueName: \"kubernetes.io/projected/9cd48d14-cd97-4894-a43e-fa678a1833b3-kube-api-access-g4xj4\") pod \"network-metrics-daemon-rj77q\" (UID: \"9cd48d14-cd97-4894-a43e-fa678a1833b3\") " pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:42:42.468181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468016    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.468181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468042    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/419b608e-bfa8-4d2b-bacd-08c286fb7707-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:42.468181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468068    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b9838bc-8604-40f5-99d5-70c240eb439d-tmp-dir\") pod \"node-resolver-5qzfj\" (UID: \"9b9838bc-8604-40f5-99d5-70c240eb439d\") " pod="openshift-dns/node-resolver-5qzfj"
Oct 30 19:42:42.468181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468095    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/419b608e-bfa8-4d2b-bacd-08c286fb7707-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:42.468181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468123    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kp8z\" (UniqueName: \"kubernetes.io/projected/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-kube-api-access-5kp8z\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.468181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468148    5958 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:42:42.468181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468159    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b9838bc-8604-40f5-99d5-70c240eb439d-tmp-dir\") pod \"node-resolver-5qzfj\" (UID: \"9b9838bc-8604-40f5-99d5-70c240eb439d\") " pod="openshift-dns/node-resolver-5qzfj"
Oct 30 19:42:42.468918 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468150    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6f405174-11f7-414c-a940-2a63c8989f3a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-p527g\" (UID: \"6f405174-11f7-414c-a940-2a63c8989f3a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p527g"
Oct 30 19:42:42.468918 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468267    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91ef5bd9-9316-418a-a989-8557a38e0058-serving-certs-ca-bundle\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:42.468918 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468291    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/93cfc69d-646b-4910-972d-04c9076c6c3c-dbus\") pod \"global-pull-secret-syncer-8t48c\" (UID: \"93cfc69d-646b-4910-972d-04c9076c6c3c\") " pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:42:42.468918 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468304    5958 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:42:42.468918 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468310    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hklwg\" (UniqueName: \"kubernetes.io/projected/0f31378b-b459-4ff0-9e72-d0d4266ab7b6-kube-api-access-hklwg\") pod \"dns-default-rjdlc\" (UID: \"0f31378b-b459-4ff0-9e72-d0d4266ab7b6\") " pod="openshift-dns/dns-default-rjdlc"
Oct 30 19:42:42.468918 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468424    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d79cb0a2-da98-4c41-a201-50755411d3c1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.468918 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468445    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/419b608e-bfa8-4d2b-bacd-08c286fb7707-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:42.468918 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468480    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21e03037-74dd-44b2-bac7-791d30cf240d-system-cni-dir\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:42:42.468918 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468498    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.468918 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468526    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a94cd52c-5a75-45c0-9669-587b56e8fc70-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wd2gl\" (UID: \"a94cd52c-5a75-45c0-9669-587b56e8fc70\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl"
Oct 30 19:42:42.468918 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468561    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/154facaf-4163-41fb-a6d9-fb04a5d8e12e-secret-metrics-server-tls\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:42.468918 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468562    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d79cb0a2-da98-4c41-a201-50755411d3c1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.468918 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468634    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/154facaf-4163-41fb-a6d9-fb04a5d8e12e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:42.468918 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468659    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-tuned\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.468918 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468675    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/023e1dd6-ffb9-48c8-a407-be580b23bddb-ovnkube-script-lib\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.468918 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468693    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21e03037-74dd-44b2-bac7-791d30cf240d-cni-binary-copy\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:42:42.468918 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468735    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.468918 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468755    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/154facaf-4163-41fb-a6d9-fb04a5d8e12e-client-ca-bundle\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:42.468918 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468773    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91ef5bd9-9316-418a-a989-8557a38e0058-metrics-client-ca\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:42.468918 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468800    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret\") pod \"global-pull-secret-syncer-8t48c\" (UID: \"93cfc69d-646b-4910-972d-04c9076c6c3c\") " pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:42:42.468918 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468826    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-tuned\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.468918 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468828    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/21e03037-74dd-44b2-bac7-791d30cf240d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:42:42.468918 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468871    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.468918 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468898    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/91ef5bd9-9316-418a-a989-8557a38e0058-telemeter-client-tls\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:42.468918 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468916    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-registration-dir\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:42:42.469567 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468934    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.469567 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.468960    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-lib-modules\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.489995 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.489963    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pcf9z\""
Oct 30 19:42:42.490136 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.490005    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:42:42.490228 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.490204    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:42:42.490295 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.490279    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:42:42.490341 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.490305    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:42:42.490413 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.490279    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:42:42.491203 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.491182    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:42:42.491853 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.491830    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:42:42.491978 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.491962    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9bk6w\""
Oct 30 19:42:42.492028 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.492019    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:42:42.492879 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.492857    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:42:42.492972 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.492883    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:42:42.493202 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.493181    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:42:42.493889 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.493866    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c4d99d2-8ba6-410c-8337-f527e98d4651-cni-binary-copy\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.494834 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.494814    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:42:42.494998 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.494984    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:42:42.497583 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.497566    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:42:42.497655 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.497625    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:42:42.497655 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.497639    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:42:42.497750 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.497689    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c4d99d2-8ba6-410c-8337-f527e98d4651-multus-daemon-config\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.497814 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.497754    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:42:42.497814 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.497756    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:42:42.497914 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.497863    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:42:42.497914 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.497890    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/023e1dd6-ffb9-48c8-a407-be580b23bddb-env-overrides\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.498037 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.498022    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:42:42.498158 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.498138    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6391e2a6-bb94-4270-9fa9-f6d35f3e433c-serviceca\") pod \"node-ca-9x594\" (UID: \"6391e2a6-bb94-4270-9fa9-f6d35f3e433c\") " pod="openshift-image-registry/node-ca-9x594"
Oct 30 19:42:42.498310 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.498292    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:42:42.498368 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.498314    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:42:42.498541 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.498526    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:42:42.498602 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.498559    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:42:42.499704 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.499685    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21e03037-74dd-44b2-bac7-791d30cf240d-cni-binary-copy\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:42:42.499765 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.499725    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/023e1dd6-ffb9-48c8-a407-be580b23bddb-ovnkube-script-lib\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.504321 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.504306    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ba9ed4cb-13c2-4ead-b549-5469e2724f6c-konnectivity-ca\") pod \"konnectivity-agent-rwbsr\" (UID: \"ba9ed4cb-13c2-4ead-b549-5469e2724f6c\") " pod="kube-system/konnectivity-agent-rwbsr"
Oct 30 19:42:42.505570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.505554    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/023e1dd6-ffb9-48c8-a407-be580b23bddb-ovnkube-config\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.508869 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.508853    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f7a30ded-2282-462e-bd34-dd4ad735a3ca-iptables-alerter-script\") pod \"iptables-alerter-dxw48\" (UID: \"f7a30ded-2282-462e-bd34-dd4ad735a3ca\") " pod="openshift-network-operator/iptables-alerter-dxw48"
Oct 30 19:42:42.542103 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.542063    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 19:42:42.546879 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.546866    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 19:42:42.547001 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.546924    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 19:42:42.547001 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.546934    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 19:42:42.547001 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.546925    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 19:42:42.547196 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.547154    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:42:42.547196 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.547153    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:42:42.547196 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.547173    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:42:42.547367 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.547214    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 19:42:42.547367 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.547224    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:42:42.547367 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.547197    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:42:42.547367 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.547172    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:42:42.547367 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.547209    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:42:42.547367 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.547156    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:42:42.547367 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.547318    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/154facaf-4163-41fb-a6d9-fb04a5d8e12e-metrics-server-audit-profiles\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:42.547708 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.547517    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 19:42:42.547708 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.547520    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 19:42:42.547708 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.547533    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 19:42:42.547708 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.547546    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 19:42:42.547708 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.547530    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:42.547708 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.547655    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.547949 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.547737    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.547949 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.547753    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 19:42:42.547949 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.547908    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 19:42:42.548088 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.547954    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/419b608e-bfa8-4d2b-bacd-08c286fb7707-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:42.548088 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.548046    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.548340 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.548322    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/419b608e-bfa8-4d2b-bacd-08c286fb7707-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:42.548863 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.548839    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/154facaf-4163-41fb-a6d9-fb04a5d8e12e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:42.549006 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.548991    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/21e03037-74dd-44b2-bac7-791d30cf240d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:42:42.549006 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.548996    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91ef5bd9-9316-418a-a989-8557a38e0058-metrics-client-ca\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:42.549084 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.549073    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.553128 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.553110    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d79cb0a2-da98-4c41-a201-50755411d3c1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.553177 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.553162    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f31378b-b459-4ff0-9e72-d0d4266ab7b6-config-volume\") pod \"dns-default-rjdlc\" (UID: \"0f31378b-b459-4ff0-9e72-d0d4266ab7b6\") " pod="openshift-dns/dns-default-rjdlc"
Oct 30 19:42:42.554293 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.554275    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/56bbeb6c-dce1-42dc-9291-615c39153a7d-metrics-client-ca\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:42.554338 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.554275    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/21e03037-74dd-44b2-bac7-791d30cf240d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:42:42.555592 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.555573    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.556779 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.556761    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-accelerators-collector-config\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:42.557911 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.557896    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a94cd52c-5a75-45c0-9669-587b56e8fc70-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wd2gl\" (UID: \"a94cd52c-5a75-45c0-9669-587b56e8fc70\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl"
Oct 30 19:42:42.557959 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.557939    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25a55956-200b-4ca6-ad5e-050caf7116e4-metrics-client-ca\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:42.570084 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570060    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-var-lib-openvswitch\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.570181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570091    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-node-log\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.570181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570110    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6391e2a6-bb94-4270-9fa9-f6d35f3e433c-host\") pod \"node-ca-9x594\" (UID: \"6391e2a6-bb94-4270-9fa9-f6d35f3e433c\") " pod="openshift-image-registry/node-ca-9x594"
Oct 30 19:42:42.570181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570131    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-hostroot\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.570181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570156    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-var-lib-openvswitch\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.570181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570170    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6391e2a6-bb94-4270-9fa9-f6d35f3e433c-host\") pod \"node-ca-9x594\" (UID: \"6391e2a6-bb94-4270-9fa9-f6d35f3e433c\") " pod="openshift-image-registry/node-ca-9x594"
Oct 30 19:42:42.570181 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570159    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-node-log\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.570355 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570185    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-hostroot\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.570355 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570171    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/93cfc69d-646b-4910-972d-04c9076c6c3c-dbus\") pod \"global-pull-secret-syncer-8t48c\" (UID: \"93cfc69d-646b-4910-972d-04c9076c6c3c\") " pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:42:42.570355 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570220    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/93cfc69d-646b-4910-972d-04c9076c6c3c-dbus\") pod \"global-pull-secret-syncer-8t48c\" (UID: \"93cfc69d-646b-4910-972d-04c9076c6c3c\") " pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:42:42.570355 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570226    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21e03037-74dd-44b2-bac7-791d30cf240d-system-cni-dir\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:42:42.570355 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570256    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-registration-dir\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:42:42.570355 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570272    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-lib-modules\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.570355 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570274    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21e03037-74dd-44b2-bac7-791d30cf240d-system-cni-dir\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:42:42.570355 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570287    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-run-openvswitch\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.570355 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570306    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-sysconfig\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.570355 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570309    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-registration-dir\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:42:42.570355 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570323    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-run\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.570355 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570338    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-cnibin\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.570355 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570348    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-run-openvswitch\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.570355 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570355    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/56bbeb6c-dce1-42dc-9291-615c39153a7d-root\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570365    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-sysconfig\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570372    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-run-multus-certs\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570388    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-etc-selinux\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570395    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-lib-modules\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570397    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-run\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570413    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21e03037-74dd-44b2-bac7-791d30cf240d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570431    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/56bbeb6c-dce1-42dc-9291-615c39153a7d-root\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570433    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-run-multus-certs\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570413    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-cnibin\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570432    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9b9838bc-8604-40f5-99d5-70c240eb439d-hosts-file\") pod \"node-resolver-5qzfj\" (UID: \"9b9838bc-8604-40f5-99d5-70c240eb439d\") " pod="openshift-dns/node-resolver-5qzfj"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570473    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21e03037-74dd-44b2-bac7-791d30cf240d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570484    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570492    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9b9838bc-8604-40f5-99d5-70c240eb439d-hosts-file\") pod \"node-resolver-5qzfj\" (UID: \"9b9838bc-8604-40f5-99d5-70c240eb439d\") " pod="openshift-dns/node-resolver-5qzfj"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570495    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-etc-selinux\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570509    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570507    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-sysctl-conf\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570548    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-multus-cni-dir\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570567    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-var-lib-kubelet\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570585    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-crio-socket\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570597    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-multus-cni-dir\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570600    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-etc-openvswitch\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570615    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-run-ovn\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570628    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-crio-socket\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570633    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-run-netns\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570637    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-sysctl-conf\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570649    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-etc-openvswitch\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570655    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-run-ovn\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570629    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-var-lib-kubelet\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570675    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-run-netns\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570683    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21e03037-74dd-44b2-bac7-791d30cf240d-cnibin\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570703    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-socket-dir\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570727    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-device-dir\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570732    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21e03037-74dd-44b2-bac7-791d30cf240d-cnibin\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570754    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-sys\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570761    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-socket-dir\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570782    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-multus-socket-dir-parent\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570789    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-device-dir\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570809    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-sys\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570809    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-var-lib-cni-multus\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570836    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-multus-socket-dir-parent\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570840    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-var-lib-cni-multus\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570847    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-host\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570867    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-run-systemd\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570883    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-sys-fs\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570899    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/93cfc69d-646b-4910-972d-04c9076c6c3c-kubelet-config\") pod \"global-pull-secret-syncer-8t48c\" (UID: \"93cfc69d-646b-4910-972d-04c9076c6c3c\") " pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570902    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-host\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.570895 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570924    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-run-systemd\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570935    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/93cfc69d-646b-4910-972d-04c9076c6c3c-kubelet-config\") pod \"global-pull-secret-syncer-8t48c\" (UID: \"93cfc69d-646b-4910-972d-04c9076c6c3c\") " pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570934    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/08a234af-5c01-4901-80ae-2ec70fa08bfe-sys-fs\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570944    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56bbeb6c-dce1-42dc-9291-615c39153a7d-sys\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570961    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56bbeb6c-dce1-42dc-9291-615c39153a7d-sys\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570978    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-sysctl-d\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.570995    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-system-cni-dir\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571019    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-cni-netd\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571040    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-var-lib-kubelet\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571056    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-kubelet\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571077    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-multus-conf-dir\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571094    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-wtmp\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571120    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-cni-netd\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571137    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-kubelet\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571144    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-multus-conf-dir\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571112    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-var-lib-kubelet\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571145    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-system-cni-dir\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571140    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-systemd-units\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571191    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-log-socket\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571194    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-sysctl-d\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571213    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-wtmp\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571224    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-cni-bin\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571234    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-systemd-units\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571242    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-var-lib-cni-bin\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571256    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-log-socket\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571260    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-kubernetes\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571267    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-cni-bin\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571292    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-var-lib-cni-bin\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571299    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-slash\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571318    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-kubernetes\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571321    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-slash\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571331    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-run-netns\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571371    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-systemd\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571402    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-run-k8s-cni-cncf-io\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571405    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-run-netns\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571430    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-systemd\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571436    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-etc-kubernetes\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571422    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-host-run-k8s-cni-cncf-io\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571484    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c4d99d2-8ba6-410c-8337-f527e98d4651-etc-kubernetes\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571497    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7a30ded-2282-462e-bd34-dd4ad735a3ca-host-slash\") pod \"iptables-alerter-dxw48\" (UID: \"f7a30ded-2282-462e-bd34-dd4ad735a3ca\") " pod="openshift-network-operator/iptables-alerter-dxw48"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571527    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-modprobe-d\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571556    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571573    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7a30ded-2282-462e-bd34-dd4ad735a3ca-host-slash\") pod \"iptables-alerter-dxw48\" (UID: \"f7a30ded-2282-462e-bd34-dd4ad735a3ca\") " pod="openshift-network-operator/iptables-alerter-dxw48"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571592    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-run-ovn-kubernetes\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571613    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571617    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/043ccaed-b00b-4406-b156-eb95ea8e05f3-etc-modprobe-d\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.572484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.571687    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/023e1dd6-ffb9-48c8-a407-be580b23bddb-host-run-ovn-kubernetes\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.598227 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.598136    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 19:42:42.598227 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.598137    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 19:42:42.598408 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.598228    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 19:42:42.598490 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.598465    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 19:42:42.598877 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.598850    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 19:42:42.598877 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.598859    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhn8m\" (UniqueName: \"kubernetes.io/projected/56bbeb6c-dce1-42dc-9291-615c39153a7d-kube-api-access-hhn8m\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:42.599035 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.598904    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cktlc\" (UniqueName: \"kubernetes.io/projected/f7a30ded-2282-462e-bd34-dd4ad735a3ca-kube-api-access-cktlc\") pod \"iptables-alerter-dxw48\" (UID: \"f7a30ded-2282-462e-bd34-dd4ad735a3ca\") " pod="openshift-network-operator/iptables-alerter-dxw48"
Oct 30 19:42:42.599035 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.598971    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91ef5bd9-9316-418a-a989-8557a38e0058-serving-certs-ca-bundle\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:42.599249 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.599225    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.599357 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.599325    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j7rs\" (UniqueName: \"kubernetes.io/projected/419b608e-bfa8-4d2b-bacd-08c286fb7707-kube-api-access-4j7rs\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:42.599422 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.599346    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzjng\" (UniqueName: \"kubernetes.io/projected/5afcafb9-baea-4a5f-906a-1ac08f1f6834-kube-api-access-qzjng\") pod \"ingress-canary-b7jpb\" (UID: \"5afcafb9-baea-4a5f-906a-1ac08f1f6834\") " pod="openshift-ingress-canary/ingress-canary-b7jpb"
Oct 30 19:42:42.599535 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.599515    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdwx2\" (UniqueName: \"kubernetes.io/projected/9b9838bc-8604-40f5-99d5-70c240eb439d-kube-api-access-cdwx2\") pod \"node-resolver-5qzfj\" (UID: \"9b9838bc-8604-40f5-99d5-70c240eb439d\") " pod="openshift-dns/node-resolver-5qzfj"
Oct 30 19:42:42.599618 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.599594    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5ngc\" (UniqueName: \"kubernetes.io/projected/08a234af-5c01-4901-80ae-2ec70fa08bfe-kube-api-access-g5ngc\") pod \"aws-ebs-csi-driver-node-prb52\" (UID: \"08a234af-5c01-4901-80ae-2ec70fa08bfe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-prb52"
Oct 30 19:42:42.599797 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.599778    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:42:42.599987 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.599967    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 19:42:42.600068 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.600051    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 19:42:42.600131 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.600112    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2tsl\" (UniqueName: \"kubernetes.io/projected/14e5d4c8-c53b-43b7-8301-804226ba1867-kube-api-access-v2tsl\") pod \"network-check-target-m9t6w\" (UID: \"14e5d4c8-c53b-43b7-8301-804226ba1867\") " pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:42:42.600131 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.600126    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 19:42:42.600224 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.600144    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49wqq\" (UniqueName: \"kubernetes.io/projected/6391e2a6-bb94-4270-9fa9-f6d35f3e433c-kube-api-access-49wqq\") pod \"node-ca-9x594\" (UID: \"6391e2a6-bb94-4270-9fa9-f6d35f3e433c\") " pod="openshift-image-registry/node-ca-9x594"
Oct 30 19:42:42.600224 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.600190    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 19:42:42.600319 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.600260    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 19:42:42.600319 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.600273    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4xj4\" (UniqueName: \"kubernetes.io/projected/9cd48d14-cd97-4894-a43e-fa678a1833b3-kube-api-access-g4xj4\") pod \"network-metrics-daemon-rj77q\" (UID: \"9cd48d14-cd97-4894-a43e-fa678a1833b3\") " pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:42:42.600319 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.600291    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n9wq\" (UniqueName: \"kubernetes.io/projected/91ef5bd9-9316-418a-a989-8557a38e0058-kube-api-access-7n9wq\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:42.600319 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.600306    5958 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:42:42.600544 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.600330    5958 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:42:42.600544 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.600344    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 19:42:42.600544 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.600404    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 19:42:42.600544 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.600444    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 19:42:42.600544 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.600475    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5mbb\" (UniqueName: \"kubernetes.io/projected/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-kube-api-access-c5mbb\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:42.600544 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.600492    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 19:42:42.600544 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.600344    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 19:42:42.600544 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.600509    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 19:42:42.600544 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.600399    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.600957 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.600555    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 19:42:42.600957 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.600558    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 19:42:42.600957 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.600572    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 19:42:42.600957 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:42:42.600681    5958 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:42:42.600957 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.600791    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 19:42:42.600957 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.600833    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk87g\" (UniqueName: \"kubernetes.io/projected/21e03037-74dd-44b2-bac7-791d30cf240d-kube-api-access-hk87g\") pod \"multus-additional-cni-plugins-snlg9\" (UID: \"21e03037-74dd-44b2-bac7-791d30cf240d\") " pod="openshift-multus/multus-additional-cni-plugins-snlg9"
Oct 30 19:42:42.600957 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.600864    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 19:42:42.600957 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.600901    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 19:42:42.601336 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.601025    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85gjj\" (UniqueName: \"kubernetes.io/projected/154facaf-4163-41fb-a6d9-fb04a5d8e12e-kube-api-access-85gjj\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:42.601336 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:42:42.601047    5958 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal"
Oct 30 19:42:42.601336 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.601169    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td8bh\" (UniqueName: \"kubernetes.io/projected/023e1dd6-ffb9-48c8-a407-be580b23bddb-kube-api-access-td8bh\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.601336 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.601304    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8b99\" (UniqueName: \"kubernetes.io/projected/d79cb0a2-da98-4c41-a201-50755411d3c1-kube-api-access-v8b99\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.601336 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.601334    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-shc6v\""
Oct 30 19:42:42.601593 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.601347    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4drl\" (UniqueName: \"kubernetes.io/projected/a94cd52c-5a75-45c0-9669-587b56e8fc70-kube-api-access-k4drl\") pod \"openshift-state-metrics-7bd7489957-wd2gl\" (UID: \"a94cd52c-5a75-45c0-9669-587b56e8fc70\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl"
Oct 30 19:42:42.601593 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.601378    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-5r6h9\""
Oct 30 19:42:42.601593 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.601433    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tqb6z\""
Oct 30 19:42:42.601593 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.601475    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hklwg\" (UniqueName: \"kubernetes.io/projected/0f31378b-b459-4ff0-9e72-d0d4266ab7b6-kube-api-access-hklwg\") pod \"dns-default-rjdlc\" (UID: \"0f31378b-b459-4ff0-9e72-d0d4266ab7b6\") " pod="openshift-dns/dns-default-rjdlc"
Oct 30 19:42:42.601784 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.601600    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:42:42.601784 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.601744    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgm8w\" (UniqueName: \"kubernetes.io/projected/6f405174-11f7-414c-a940-2a63c8989f3a-kube-api-access-dgm8w\") pod \"monitoring-plugin-77f4f76594-p527g\" (UID: \"6f405174-11f7-414c-a940-2a63c8989f3a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p527g"
Oct 30 19:42:42.601885 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.601788    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s45zp\" (UniqueName: \"kubernetes.io/projected/7cf2adf5-da5b-4d63-a490-9e88f3598288-kube-api-access-s45zp\") pod \"downloads-6dd789c8b7-zk9rp\" (UID: \"7cf2adf5-da5b-4d63-a490-9e88f3598288\") " pod="openshift-console/downloads-6dd789c8b7-zk9rp"
Oct 30 19:42:42.601885 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.601794    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g28pf\" (UniqueName: \"kubernetes.io/projected/043ccaed-b00b-4406-b156-eb95ea8e05f3-kube-api-access-g28pf\") pod \"tuned-qz2rj\" (UID: \"043ccaed-b00b-4406-b156-eb95ea8e05f3\") " pod="openshift-cluster-node-tuning-operator/tuned-qz2rj"
Oct 30 19:42:42.601885 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.601788    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc48c\" (UniqueName: \"kubernetes.io/projected/1c4d99d2-8ba6-410c-8337-f527e98d4651-kube-api-access-rc48c\") pod \"multus-l2mtk\" (UID: \"1c4d99d2-8ba6-410c-8337-f527e98d4651\") " pod="openshift-multus/multus-l2mtk"
Oct 30 19:42:42.602041 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.601973    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kp8z\" (UniqueName: \"kubernetes.io/projected/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-kube-api-access-5kp8z\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.602215 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.602195    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbcq7\" (UniqueName: \"kubernetes.io/projected/25a55956-200b-4ca6-ad5e-050caf7116e4-kube-api-access-nbcq7\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:42.603957 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.603937    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5afcafb9-baea-4a5f-906a-1ac08f1f6834-cert\") pod \"ingress-canary-b7jpb\" (UID: \"5afcafb9-baea-4a5f-906a-1ac08f1f6834\") " pod="openshift-ingress-canary/ingress-canary-b7jpb"
Oct 30 19:42:42.604029 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.603986    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d79cb0a2-da98-4c41-a201-50755411d3c1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.604029 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.604022    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91ef5bd9-9316-418a-a989-8557a38e0058-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:42.604992 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.604810    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 19:42:42.604992 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.604816    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-tbtn8\""
Oct 30 19:42:42.604992 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.604832    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 19:42:42.604992 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.604855    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 19:42:42.604992 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.604885    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/91ef5bd9-9316-418a-a989-8557a38e0058-federate-client-tls\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:42.604992 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.604895    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 19:42:42.604992 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.604942    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.604992 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.604953    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d17vqncs64e3\""
Oct 30 19:42:42.604992 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.604962    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 19:42:42.604992 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.604970    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 19:42:42.604992 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.604972    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 19:42:42.604992 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.604969    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605061    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-47f99\""
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605086    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2j2tt\""
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605094    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d79cb0a2-da98-4c41-a201-50755411d3c1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605108    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j4jh7\""
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605086    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605165    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605184    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605199    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7t1rbpjevshnn\""
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605212    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-7t4pl\""
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605251    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605270    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605291    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-v5m2b\""
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605294    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605288    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605314    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-vlbmc\""
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605254    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605340    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605362    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605345    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605373    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605318    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605343    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605407    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605416    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605490    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605541    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.605570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.605548    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.606285 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.606241    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 19:42:42.606329 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.606310    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:42.606359 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.606337    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-f0ill0u6gh1v0\""
Oct 30 19:42:42.606359 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.606336    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/154facaf-4163-41fb-a6d9-fb04a5d8e12e-secret-metrics-server-client-certs\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:42.607560 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.607527    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.607560 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.607545    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.607687 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.607552    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/91ef5bd9-9316-418a-a989-8557a38e0058-secret-telemeter-client\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:42.607687 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.607585    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-grpc-tls\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:42.607687 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.607650    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.607687 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.607677    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a94cd52c-5a75-45c0-9669-587b56e8fc70-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wd2gl\" (UID: \"a94cd52c-5a75-45c0-9669-587b56e8fc70\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl"
Oct 30 19:42:42.607835 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.607692    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-vxj8j\""
Oct 30 19:42:42.607835 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.607711    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-config-volume\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.608543 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.608303    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/023e1dd6-ffb9-48c8-a407-be580b23bddb-ovn-node-metrics-cert\") pod \"ovnkube-node-ft6pg\" (UID: \"023e1dd6-ffb9-48c8-a407-be580b23bddb\") " pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:42.608543 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.608314    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:42.608543 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.608341    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tcwj6\""
Oct 30 19:42:42.608543 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.608368    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pxncz\""
Oct 30 19:42:42.608543 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.608373    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.608543 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.608408    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6f405174-11f7-414c-a940-2a63c8989f3a-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-p527g\" (UID: \"6f405174-11f7-414c-a940-2a63c8989f3a\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-p527g"
Oct 30 19:42:42.608543 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.608413    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fn2xq\""
Oct 30 19:42:42.608543 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.608431    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 19:42:42.608543 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.608502    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91ef5bd9-9316-418a-a989-8557a38e0058-telemeter-trusted-ca-bundle\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:42.608953 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.608563    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hsfbm\""
Oct 30 19:42:42.608953 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.608562    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/419b608e-bfa8-4d2b-bacd-08c286fb7707-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:42.608953 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.608703    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pjk7j\""
Oct 30 19:42:42.608953 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.608846    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.609130 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.609005    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2zvp8\""
Oct 30 19:42:42.609130 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.609109    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/419b608e-bfa8-4d2b-bacd-08c286fb7707-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-56t97\" (UID: \"419b608e-bfa8-4d2b-bacd-08c286fb7707\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-56t97"
Oct 30 19:42:42.609219 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.609188    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:42:42.609219 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.609209    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a94cd52c-5a75-45c0-9669-587b56e8fc70-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wd2gl\" (UID: \"a94cd52c-5a75-45c0-9669-587b56e8fc70\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wd2gl"
Oct 30 19:42:42.612288 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.609354    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/154facaf-4163-41fb-a6d9-fb04a5d8e12e-client-ca-bundle\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:42.612288 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.609362    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/91ef5bd9-9316-418a-a989-8557a38e0058-telemeter-client-tls\") pod \"telemeter-client-596848ccc9-tsm2b\" (UID: \"91ef5bd9-9316-418a-a989-8557a38e0058\") " pod="openshift-monitoring/telemeter-client-596848ccc9-tsm2b"
Oct 30 19:42:42.612288 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.609359    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 19:42:42.612288 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.609402    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-x99sr\""
Oct 30 19:42:42.612288 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.609538    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-v9l6w\""
Oct 30 19:42:42.612288 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.609604    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6dhxv\""
Oct 30 19:42:42.612288 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.609688    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:42:42.612288 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.609719    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd48d14-cd97-4894-a43e-fa678a1833b3-metrics-certs\") pod \"network-metrics-daemon-rj77q\" (UID: \"9cd48d14-cd97-4894-a43e-fa678a1833b3\") " pod="openshift-multus/network-metrics-daemon-rj77q"
Oct 30 19:42:42.612288 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.609862    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qgpv7\""
Oct 30 19:42:42.612288 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.609866    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2wfbg\""
Oct 30 19:42:42.612288 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.609975    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.612288 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.610041    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/154facaf-4163-41fb-a6d9-fb04a5d8e12e-secret-metrics-server-tls\") pod \"metrics-server-8698dc55b4-pwqbt\" (UID: \"154facaf-4163-41fb-a6d9-fb04a5d8e12e\") " pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:42:42.614566 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.614543    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-config\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.614566 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.614557    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-web-config\") pod \"prometheus-k8s-0\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:42.614686 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.614552    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-web-config\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.614686 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.614579    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:42:42.614686 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.614600    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/25a55956-200b-4ca6-ad5e-050caf7116e4-secret-thanos-querier-tls\") pod \"thanos-querier-5f5bcd9b49-9bcsl\" (UID: \"25a55956-200b-4ca6-ad5e-050caf7116e4\") " pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:42.614686 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.614660    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b81aa0b-b68a-4225-81e6-0e5d59d5c63b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t4b2n\" (UID: \"0b81aa0b-b68a-4225-81e6-0e5d59d5c63b\") " pod="openshift-insights/insights-runtime-extractor-t4b2n"
Oct 30 19:42:42.615720 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.615705    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/56bbeb6c-dce1-42dc-9291-615c39153a7d-node-exporter-tls\") pod \"node-exporter-t586q\" (UID: \"56bbeb6c-dce1-42dc-9291-615c39153a7d\") " pod="openshift-monitoring/node-exporter-t586q"
Oct 30 19:42:42.615827 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.615806    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ba9ed4cb-13c2-4ead-b549-5469e2724f6c-agent-certs\") pod \"konnectivity-agent-rwbsr\" (UID: \"ba9ed4cb-13c2-4ead-b549-5469e2724f6c\") " pod="kube-system/konnectivity-agent-rwbsr"
Oct 30 19:42:42.615827 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.615824    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f31378b-b459-4ff0-9e72-d0d4266ab7b6-metrics-tls\") pod \"dns-default-rjdlc\" (UID: \"0f31378b-b459-4ff0-9e72-d0d4266ab7b6\") " pod="openshift-dns/dns-default-rjdlc"
Oct 30 19:42:42.620131 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.620115    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93cfc69d-646b-4910-972d-04c9076c6c3c-original-pull-secret\") pod \"global-pull-secret-syncer-8t48c\" (UID: \"93cfc69d-646b-4910-972d-04c9076c6c3c\") " pod="kube-system/global-pull-secret-syncer-8t48c"
Oct 30 19:42:42.749883 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:42.749861    5958 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:42:43.347031 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:43.346992    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rjdlc"
Oct 30 19:42:43.526001 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:43.525967    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5f5bcd9b49-9bcsl"
Oct 30 19:42:44.980083 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:44.980044    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-p527g"
Oct 30 19:42:44.981601 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:44.981562    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:45.027275 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:45.027249    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rwbsr"
Oct 30 19:42:45.038325 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:45.038297    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:42:47.134144 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:47.134104    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:42:48.272585 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:48.272553    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:52.438862 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:52.438224    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:42:52.438862 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:42:52.438508    5958 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:43:00.517631 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:00.517597    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:43:00.518106 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:00.518041    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="kube-rbac-proxy" containerID="cri-o://d02c58ca04cf1da1a94cb78ca4de61b4414708f8412a6284d2ac074667676deb" gracePeriod=120
Oct 30 19:43:00.518106 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:00.518096    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="config-reloader" containerID="cri-o://89cc8f3dfe90249695ac5f814cc9ef56147bad383ba9c49f3c4e8cfab4c83dad" gracePeriod=120
Oct 30 19:43:00.518226 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:00.518034    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="alertmanager" containerID="cri-o://f1e0f2582e88ef4830f08b643b1a96719eee6f9522f5859cf09355864a55a779" gracePeriod=120
Oct 30 19:43:00.518226 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:00.518148    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="prom-label-proxy" containerID="cri-o://e05fe4e8501643275a673151999aea5e8e64bef7c88bb8ce70c18496da9f9bae" gracePeriod=120
Oct 30 19:43:00.518328 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:00.518214    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="kube-rbac-proxy-metric" containerID="cri-o://054a4388d3ec6bc2e9471fa4f2964f309d0cf36ebdc908990a21850c43b6aca2" gracePeriod=120
Oct 30 19:43:00.518328 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:00.518081    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="kube-rbac-proxy-web" containerID="cri-o://a53d819b93dfeb0f7311cdfbdfee03f619136b2796c1c299632ddb2fc2882fc2" gracePeriod=120
Oct 30 19:43:00.552693 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:00.552664    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-zk9rp" event={"ID":"7cf2adf5-da5b-4d63-a490-9e88f3598288","Type":"ContainerStarted","Data":"8f1ed7298847e635b17ed59ccff7ab8b8c6e9925a1c8ef52ed3f8ddb8f910f3b"}
Oct 30 19:43:00.552976 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:00.552949    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-zk9rp"
Oct 30 19:43:00.554352 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:00.554328    5958 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-zk9rp container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.13:8080/\": dial tcp 10.134.0.13:8080: connect: connection refused" start-of-body=
Oct 30 19:43:00.554412 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:00.554384    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-zk9rp" podUID="7cf2adf5-da5b-4d63-a490-9e88f3598288" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.13:8080/\": dial tcp 10.134.0.13:8080: connect: connection refused"
Oct 30 19:43:01.580209 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.580172    5958 generic.go:356] "Generic (PLEG): container finished" podID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerID="e05fe4e8501643275a673151999aea5e8e64bef7c88bb8ce70c18496da9f9bae" exitCode=0
Oct 30 19:43:01.580209 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.580206    5958 generic.go:356] "Generic (PLEG): container finished" podID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerID="054a4388d3ec6bc2e9471fa4f2964f309d0cf36ebdc908990a21850c43b6aca2" exitCode=0
Oct 30 19:43:01.580209 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.580212    5958 generic.go:356] "Generic (PLEG): container finished" podID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerID="d02c58ca04cf1da1a94cb78ca4de61b4414708f8412a6284d2ac074667676deb" exitCode=0
Oct 30 19:43:01.580882 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.580218    5958 generic.go:356] "Generic (PLEG): container finished" podID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerID="89cc8f3dfe90249695ac5f814cc9ef56147bad383ba9c49f3c4e8cfab4c83dad" exitCode=0
Oct 30 19:43:01.580882 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.580223    5958 generic.go:356] "Generic (PLEG): container finished" podID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerID="f1e0f2582e88ef4830f08b643b1a96719eee6f9522f5859cf09355864a55a779" exitCode=0
Oct 30 19:43:01.580882 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.580309    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d79cb0a2-da98-4c41-a201-50755411d3c1","Type":"ContainerDied","Data":"e05fe4e8501643275a673151999aea5e8e64bef7c88bb8ce70c18496da9f9bae"}
Oct 30 19:43:01.580882 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.580405    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d79cb0a2-da98-4c41-a201-50755411d3c1","Type":"ContainerDied","Data":"054a4388d3ec6bc2e9471fa4f2964f309d0cf36ebdc908990a21850c43b6aca2"}
Oct 30 19:43:01.580882 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.580483    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d79cb0a2-da98-4c41-a201-50755411d3c1","Type":"ContainerDied","Data":"d02c58ca04cf1da1a94cb78ca4de61b4414708f8412a6284d2ac074667676deb"}
Oct 30 19:43:01.580882 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.580529    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d79cb0a2-da98-4c41-a201-50755411d3c1","Type":"ContainerDied","Data":"89cc8f3dfe90249695ac5f814cc9ef56147bad383ba9c49f3c4e8cfab4c83dad"}
Oct 30 19:43:01.580882 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.580570    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d79cb0a2-da98-4c41-a201-50755411d3c1","Type":"ContainerDied","Data":"f1e0f2582e88ef4830f08b643b1a96719eee6f9522f5859cf09355864a55a779"}
Oct 30 19:43:01.592885 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.592782    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-zk9rp"
Oct 30 19:43:01.820982 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.820953    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:01.959445 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.959348    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-web-config\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-web-config\") pod \"d79cb0a2-da98-4c41-a201-50755411d3c1\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") "
Oct 30 19:43:01.959655 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.959468    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-config-volume\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-config-volume\") pod \"d79cb0a2-da98-4c41-a201-50755411d3c1\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") "
Oct 30 19:43:01.959655 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.959505    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d79cb0a2-da98-4c41-a201-50755411d3c1-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d79cb0a2-da98-4c41-a201-50755411d3c1-metrics-client-ca\") pod \"d79cb0a2-da98-4c41-a201-50755411d3c1\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") "
Oct 30 19:43:01.959655 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.959532    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy\") pod \"d79cb0a2-da98-4c41-a201-50755411d3c1\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") "
Oct 30 19:43:01.959655 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.959568    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d79cb0a2-da98-4c41-a201-50755411d3c1-kube-api-access-v8b99\" (UniqueName: \"kubernetes.io/projected/d79cb0a2-da98-4c41-a201-50755411d3c1-kube-api-access-v8b99\") pod \"d79cb0a2-da98-4c41-a201-50755411d3c1\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") "
Oct 30 19:43:01.959655 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.959605    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d79cb0a2-da98-4c41-a201-50755411d3c1-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d79cb0a2-da98-4c41-a201-50755411d3c1-alertmanager-trusted-ca-bundle\") pod \"d79cb0a2-da98-4c41-a201-50755411d3c1\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") "
Oct 30 19:43:01.959655 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.959644    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-main-tls\") pod \"d79cb0a2-da98-4c41-a201-50755411d3c1\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") "
Oct 30 19:43:01.959971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.959686    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d79cb0a2-da98-4c41-a201-50755411d3c1-config-out\" (UniqueName: \"kubernetes.io/empty-dir/d79cb0a2-da98-4c41-a201-50755411d3c1-config-out\") pod \"d79cb0a2-da98-4c41-a201-50755411d3c1\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") "
Oct 30 19:43:01.959971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.959727    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy-web\") pod \"d79cb0a2-da98-4c41-a201-50755411d3c1\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") "
Oct 30 19:43:01.959971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.959759    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-cluster-tls-config\") pod \"d79cb0a2-da98-4c41-a201-50755411d3c1\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") "
Oct 30 19:43:01.959971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.959799    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d79cb0a2-da98-4c41-a201-50755411d3c1-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d79cb0a2-da98-4c41-a201-50755411d3c1-alertmanager-main-db\") pod \"d79cb0a2-da98-4c41-a201-50755411d3c1\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") "
Oct 30 19:43:01.959971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.959838    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d79cb0a2-da98-4c41-a201-50755411d3c1-tls-assets\" (UniqueName: \"kubernetes.io/projected/d79cb0a2-da98-4c41-a201-50755411d3c1-tls-assets\") pod \"d79cb0a2-da98-4c41-a201-50755411d3c1\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") "
Oct 30 19:43:01.959971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.959869    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d79cb0a2-da98-4c41-a201-50755411d3c1\" (UID: \"d79cb0a2-da98-4c41-a201-50755411d3c1\") "
Oct 30 19:43:01.959971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.959878    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d79cb0a2-da98-4c41-a201-50755411d3c1-metrics-client-ca" pod "d79cb0a2-da98-4c41-a201-50755411d3c1" (UID: "d79cb0a2-da98-4c41-a201-50755411d3c1"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:01.960318 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.960132    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d79cb0a2-da98-4c41-a201-50755411d3c1-metrics-client-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:01.961545 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.961510    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79cb0a2-da98-4c41-a201-50755411d3c1-alertmanager-main-db" pod "d79cb0a2-da98-4c41-a201-50755411d3c1" (UID: "d79cb0a2-da98-4c41-a201-50755411d3c1"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:01.964036 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.964006    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d79cb0a2-da98-4c41-a201-50755411d3c1-alertmanager-trusted-ca-bundle" pod "d79cb0a2-da98-4c41-a201-50755411d3c1" (UID: "d79cb0a2-da98-4c41-a201-50755411d3c1"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:01.964528 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.964498    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-config-volume" pod "d79cb0a2-da98-4c41-a201-50755411d3c1" (UID: "d79cb0a2-da98-4c41-a201-50755411d3c1"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:01.964887 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.964693    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d79cb0a2-da98-4c41-a201-50755411d3c1-kube-api-access-v8b99" pod "d79cb0a2-da98-4c41-a201-50755411d3c1" (UID: "d79cb0a2-da98-4c41-a201-50755411d3c1"). InnerVolumeSpecName "kube-api-access-v8b99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:01.965751 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.965719    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy-metric" pod "d79cb0a2-da98-4c41-a201-50755411d3c1" (UID: "d79cb0a2-da98-4c41-a201-50755411d3c1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:01.965836 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.965746    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d79cb0a2-da98-4c41-a201-50755411d3c1-tls-assets" pod "d79cb0a2-da98-4c41-a201-50755411d3c1" (UID: "d79cb0a2-da98-4c41-a201-50755411d3c1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:01.965836 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.965781    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-main-tls" pod "d79cb0a2-da98-4c41-a201-50755411d3c1" (UID: "d79cb0a2-da98-4c41-a201-50755411d3c1"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:01.966233 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.966213    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy" pod "d79cb0a2-da98-4c41-a201-50755411d3c1" (UID: "d79cb0a2-da98-4c41-a201-50755411d3c1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:01.966586 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.966559    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy-web" pod "d79cb0a2-da98-4c41-a201-50755411d3c1" (UID: "d79cb0a2-da98-4c41-a201-50755411d3c1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:01.967373 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.967348    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79cb0a2-da98-4c41-a201-50755411d3c1-config-out" pod "d79cb0a2-da98-4c41-a201-50755411d3c1" (UID: "d79cb0a2-da98-4c41-a201-50755411d3c1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:01.971549 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.971519    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-cluster-tls-config" pod "d79cb0a2-da98-4c41-a201-50755411d3c1" (UID: "d79cb0a2-da98-4c41-a201-50755411d3c1"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:01.979121 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:01.979075    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-web-config" pod "d79cb0a2-da98-4c41-a201-50755411d3c1" (UID: "d79cb0a2-da98-4c41-a201-50755411d3c1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:02.061105 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.060972    5958 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:02.061105 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.061014    5958 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:02.061105 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.061032    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8b99\" (UniqueName: \"kubernetes.io/projected/d79cb0a2-da98-4c41-a201-50755411d3c1-kube-api-access-v8b99\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:02.061105 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.061048    5958 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d79cb0a2-da98-4c41-a201-50755411d3c1-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:02.061105 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.061064    5958 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-main-tls\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:02.061105 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.061077    5958 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d79cb0a2-da98-4c41-a201-50755411d3c1-config-out\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:02.061105 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.061091    5958 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:02.061105 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.061105    5958 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-cluster-tls-config\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:02.061105 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.061118    5958 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d79cb0a2-da98-4c41-a201-50755411d3c1-alertmanager-main-db\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:02.061726 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.061131    5958 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d79cb0a2-da98-4c41-a201-50755411d3c1-tls-assets\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:02.061726 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.061145    5958 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:02.061726 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.061159    5958 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d79cb0a2-da98-4c41-a201-50755411d3c1-web-config\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:02.589401 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.589365    5958 generic.go:356] "Generic (PLEG): container finished" podID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerID="a53d819b93dfeb0f7311cdfbdfee03f619136b2796c1c299632ddb2fc2882fc2" exitCode=0
Oct 30 19:43:02.589846 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.589477    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:02.589846 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.589474    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d79cb0a2-da98-4c41-a201-50755411d3c1","Type":"ContainerDied","Data":"a53d819b93dfeb0f7311cdfbdfee03f619136b2796c1c299632ddb2fc2882fc2"}
Oct 30 19:43:02.589846 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.589623    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d79cb0a2-da98-4c41-a201-50755411d3c1","Type":"ContainerDied","Data":"ca771da4e671d62f4d3e26f0a503559a4c02b532afec77ed8b75626b74cbe8ea"}
Oct 30 19:43:02.589846 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.589677    5958 scope.go:122] "RemoveContainer" containerID="e05fe4e8501643275a673151999aea5e8e64bef7c88bb8ce70c18496da9f9bae"
Oct 30 19:43:02.599632 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.599513    5958 scope.go:122] "RemoveContainer" containerID="054a4388d3ec6bc2e9471fa4f2964f309d0cf36ebdc908990a21850c43b6aca2"
Oct 30 19:43:02.608267 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.608171    5958 scope.go:122] "RemoveContainer" containerID="d02c58ca04cf1da1a94cb78ca4de61b4414708f8412a6284d2ac074667676deb"
Oct 30 19:43:02.620708 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.620681    5958 scope.go:122] "RemoveContainer" containerID="a53d819b93dfeb0f7311cdfbdfee03f619136b2796c1c299632ddb2fc2882fc2"
Oct 30 19:43:02.629580 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.629558    5958 scope.go:122] "RemoveContainer" containerID="89cc8f3dfe90249695ac5f814cc9ef56147bad383ba9c49f3c4e8cfab4c83dad"
Oct 30 19:43:02.637972 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.637951    5958 scope.go:122] "RemoveContainer" containerID="f1e0f2582e88ef4830f08b643b1a96719eee6f9522f5859cf09355864a55a779"
Oct 30 19:43:02.647011 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.646985    5958 scope.go:122] "RemoveContainer" containerID="a9af2f1df48ee00d2017099c3228cbb04f43cef68b474005219ce36c44e86c10"
Oct 30 19:43:02.655925 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.655899    5958 scope.go:122] "RemoveContainer" containerID="e05fe4e8501643275a673151999aea5e8e64bef7c88bb8ce70c18496da9f9bae"
Oct 30 19:43:02.656280 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:43:02.656255    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e05fe4e8501643275a673151999aea5e8e64bef7c88bb8ce70c18496da9f9bae\": container with ID starting with e05fe4e8501643275a673151999aea5e8e64bef7c88bb8ce70c18496da9f9bae not found: ID does not exist" containerID="e05fe4e8501643275a673151999aea5e8e64bef7c88bb8ce70c18496da9f9bae"
Oct 30 19:43:02.656366 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.656291    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05fe4e8501643275a673151999aea5e8e64bef7c88bb8ce70c18496da9f9bae"} err="failed to get container status \"e05fe4e8501643275a673151999aea5e8e64bef7c88bb8ce70c18496da9f9bae\": rpc error: code = NotFound desc = could not find container \"e05fe4e8501643275a673151999aea5e8e64bef7c88bb8ce70c18496da9f9bae\": container with ID starting with e05fe4e8501643275a673151999aea5e8e64bef7c88bb8ce70c18496da9f9bae not found: ID does not exist"
Oct 30 19:43:02.656366 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.656340    5958 scope.go:122] "RemoveContainer" containerID="054a4388d3ec6bc2e9471fa4f2964f309d0cf36ebdc908990a21850c43b6aca2"
Oct 30 19:43:02.656678 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:43:02.656656    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"054a4388d3ec6bc2e9471fa4f2964f309d0cf36ebdc908990a21850c43b6aca2\": container with ID starting with 054a4388d3ec6bc2e9471fa4f2964f309d0cf36ebdc908990a21850c43b6aca2 not found: ID does not exist" containerID="054a4388d3ec6bc2e9471fa4f2964f309d0cf36ebdc908990a21850c43b6aca2"
Oct 30 19:43:02.656765 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.656687    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"054a4388d3ec6bc2e9471fa4f2964f309d0cf36ebdc908990a21850c43b6aca2"} err="failed to get container status \"054a4388d3ec6bc2e9471fa4f2964f309d0cf36ebdc908990a21850c43b6aca2\": rpc error: code = NotFound desc = could not find container \"054a4388d3ec6bc2e9471fa4f2964f309d0cf36ebdc908990a21850c43b6aca2\": container with ID starting with 054a4388d3ec6bc2e9471fa4f2964f309d0cf36ebdc908990a21850c43b6aca2 not found: ID does not exist"
Oct 30 19:43:02.656765 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.656708    5958 scope.go:122] "RemoveContainer" containerID="d02c58ca04cf1da1a94cb78ca4de61b4414708f8412a6284d2ac074667676deb"
Oct 30 19:43:02.657010 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:43:02.656993    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d02c58ca04cf1da1a94cb78ca4de61b4414708f8412a6284d2ac074667676deb\": container with ID starting with d02c58ca04cf1da1a94cb78ca4de61b4414708f8412a6284d2ac074667676deb not found: ID does not exist" containerID="d02c58ca04cf1da1a94cb78ca4de61b4414708f8412a6284d2ac074667676deb"
Oct 30 19:43:02.657083 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.657017    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d02c58ca04cf1da1a94cb78ca4de61b4414708f8412a6284d2ac074667676deb"} err="failed to get container status \"d02c58ca04cf1da1a94cb78ca4de61b4414708f8412a6284d2ac074667676deb\": rpc error: code = NotFound desc = could not find container \"d02c58ca04cf1da1a94cb78ca4de61b4414708f8412a6284d2ac074667676deb\": container with ID starting with d02c58ca04cf1da1a94cb78ca4de61b4414708f8412a6284d2ac074667676deb not found: ID does not exist"
Oct 30 19:43:02.657083 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.657036    5958 scope.go:122] "RemoveContainer" containerID="a53d819b93dfeb0f7311cdfbdfee03f619136b2796c1c299632ddb2fc2882fc2"
Oct 30 19:43:02.657339 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:43:02.657318    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a53d819b93dfeb0f7311cdfbdfee03f619136b2796c1c299632ddb2fc2882fc2\": container with ID starting with a53d819b93dfeb0f7311cdfbdfee03f619136b2796c1c299632ddb2fc2882fc2 not found: ID does not exist" containerID="a53d819b93dfeb0f7311cdfbdfee03f619136b2796c1c299632ddb2fc2882fc2"
Oct 30 19:43:02.657415 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.657345    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a53d819b93dfeb0f7311cdfbdfee03f619136b2796c1c299632ddb2fc2882fc2"} err="failed to get container status \"a53d819b93dfeb0f7311cdfbdfee03f619136b2796c1c299632ddb2fc2882fc2\": rpc error: code = NotFound desc = could not find container \"a53d819b93dfeb0f7311cdfbdfee03f619136b2796c1c299632ddb2fc2882fc2\": container with ID starting with a53d819b93dfeb0f7311cdfbdfee03f619136b2796c1c299632ddb2fc2882fc2 not found: ID does not exist"
Oct 30 19:43:02.657415 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.657365    5958 scope.go:122] "RemoveContainer" containerID="89cc8f3dfe90249695ac5f814cc9ef56147bad383ba9c49f3c4e8cfab4c83dad"
Oct 30 19:43:02.657658 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:43:02.657640    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89cc8f3dfe90249695ac5f814cc9ef56147bad383ba9c49f3c4e8cfab4c83dad\": container with ID starting with 89cc8f3dfe90249695ac5f814cc9ef56147bad383ba9c49f3c4e8cfab4c83dad not found: ID does not exist" containerID="89cc8f3dfe90249695ac5f814cc9ef56147bad383ba9c49f3c4e8cfab4c83dad"
Oct 30 19:43:02.657732 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.657665    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89cc8f3dfe90249695ac5f814cc9ef56147bad383ba9c49f3c4e8cfab4c83dad"} err="failed to get container status \"89cc8f3dfe90249695ac5f814cc9ef56147bad383ba9c49f3c4e8cfab4c83dad\": rpc error: code = NotFound desc = could not find container \"89cc8f3dfe90249695ac5f814cc9ef56147bad383ba9c49f3c4e8cfab4c83dad\": container with ID starting with 89cc8f3dfe90249695ac5f814cc9ef56147bad383ba9c49f3c4e8cfab4c83dad not found: ID does not exist"
Oct 30 19:43:02.657732 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.657684    5958 scope.go:122] "RemoveContainer" containerID="f1e0f2582e88ef4830f08b643b1a96719eee6f9522f5859cf09355864a55a779"
Oct 30 19:43:02.657934 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:43:02.657919    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1e0f2582e88ef4830f08b643b1a96719eee6f9522f5859cf09355864a55a779\": container with ID starting with f1e0f2582e88ef4830f08b643b1a96719eee6f9522f5859cf09355864a55a779 not found: ID does not exist" containerID="f1e0f2582e88ef4830f08b643b1a96719eee6f9522f5859cf09355864a55a779"
Oct 30 19:43:02.657992 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.657941    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e0f2582e88ef4830f08b643b1a96719eee6f9522f5859cf09355864a55a779"} err="failed to get container status \"f1e0f2582e88ef4830f08b643b1a96719eee6f9522f5859cf09355864a55a779\": rpc error: code = NotFound desc = could not find container \"f1e0f2582e88ef4830f08b643b1a96719eee6f9522f5859cf09355864a55a779\": container with ID starting with f1e0f2582e88ef4830f08b643b1a96719eee6f9522f5859cf09355864a55a779 not found: ID does not exist"
Oct 30 19:43:02.657992 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.657959    5958 scope.go:122] "RemoveContainer" containerID="a9af2f1df48ee00d2017099c3228cbb04f43cef68b474005219ce36c44e86c10"
Oct 30 19:43:02.658255 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:43:02.658236    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9af2f1df48ee00d2017099c3228cbb04f43cef68b474005219ce36c44e86c10\": container with ID starting with a9af2f1df48ee00d2017099c3228cbb04f43cef68b474005219ce36c44e86c10 not found: ID does not exist" containerID="a9af2f1df48ee00d2017099c3228cbb04f43cef68b474005219ce36c44e86c10"
Oct 30 19:43:02.658321 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.658263    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9af2f1df48ee00d2017099c3228cbb04f43cef68b474005219ce36c44e86c10"} err="failed to get container status \"a9af2f1df48ee00d2017099c3228cbb04f43cef68b474005219ce36c44e86c10\": rpc error: code = NotFound desc = could not find container \"a9af2f1df48ee00d2017099c3228cbb04f43cef68b474005219ce36c44e86c10\": container with ID starting with a9af2f1df48ee00d2017099c3228cbb04f43cef68b474005219ce36c44e86c10 not found: ID does not exist"
Oct 30 19:43:02.759807 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.759763    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:43:02.813007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.809833    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:43:02.813007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.810658    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:43:02.813007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.811090    5958 cpu_manager.go:414] "containerMap: removing container" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="prom-label-proxy"
Oct 30 19:43:02.813007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.811109    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="prom-label-proxy"
Oct 30 19:43:02.813007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.811127    5958 cpu_manager.go:414] "containerMap: removing container" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="config-reloader"
Oct 30 19:43:02.813007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.811133    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="config-reloader"
Oct 30 19:43:02.813007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.811146    5958 cpu_manager.go:414] "containerMap: removing container" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="kube-rbac-proxy-web"
Oct 30 19:43:02.813007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.811153    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="kube-rbac-proxy-web"
Oct 30 19:43:02.813007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.811168    5958 cpu_manager.go:414] "containerMap: removing container" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="kube-rbac-proxy"
Oct 30 19:43:02.813007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.811175    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="kube-rbac-proxy"
Oct 30 19:43:02.813007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.811187    5958 cpu_manager.go:414] "containerMap: removing container" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="alertmanager"
Oct 30 19:43:02.813007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.811194    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="alertmanager"
Oct 30 19:43:02.813007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.811216    5958 cpu_manager.go:414] "containerMap: removing container" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="kube-rbac-proxy-metric"
Oct 30 19:43:02.813007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.811223    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="kube-rbac-proxy-metric"
Oct 30 19:43:02.813007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.811244    5958 cpu_manager.go:414] "containerMap: removing container" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="init-config-reloader"
Oct 30 19:43:02.813007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.811251    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="init-config-reloader"
Oct 30 19:43:02.813007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.811321    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="init-config-reloader"
Oct 30 19:43:02.813007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.811336    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="prom-label-proxy"
Oct 30 19:43:02.813007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.811348    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="config-reloader"
Oct 30 19:43:02.813007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.811358    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="kube-rbac-proxy-web"
Oct 30 19:43:02.813007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.811367    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="kube-rbac-proxy"
Oct 30 19:43:02.813007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.811373    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="alertmanager"
Oct 30 19:43:02.813007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.811385    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" containerName="kube-rbac-proxy-metric"
Oct 30 19:43:02.835065 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.834986    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:02.889288 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.889252    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:43:02.893958 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.893370    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 19:43:02.893958 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.893476    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 19:43:02.893958 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.893529    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 19:43:02.893958 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.893599    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 19:43:02.893958 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.893656    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2zvp8\""
Oct 30 19:43:02.893958 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.893666    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 19:43:02.893958 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.893724    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 19:43:02.894312 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.893974    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 19:43:02.894312 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.894070    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 19:43:02.942596 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.942560    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 19:43:02.969832 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.969791    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f252503f-59b8-47c1-9ff8-f4f704a159d4-config-volume\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:02.970010 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.969843    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfw58\" (UniqueName: \"kubernetes.io/projected/f252503f-59b8-47c1-9ff8-f4f704a159d4-kube-api-access-pfw58\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:02.970010 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.969876    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f252503f-59b8-47c1-9ff8-f4f704a159d4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:02.970010 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.969912    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f252503f-59b8-47c1-9ff8-f4f704a159d4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:02.970010 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.969941    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f252503f-59b8-47c1-9ff8-f4f704a159d4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:02.970010 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.969972    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f252503f-59b8-47c1-9ff8-f4f704a159d4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:02.970010 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.969999    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f252503f-59b8-47c1-9ff8-f4f704a159d4-web-config\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:02.970283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.970034    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f252503f-59b8-47c1-9ff8-f4f704a159d4-config-out\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:02.970283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.970062    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f252503f-59b8-47c1-9ff8-f4f704a159d4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:02.970283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.970105    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f252503f-59b8-47c1-9ff8-f4f704a159d4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:02.970283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.970143    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f252503f-59b8-47c1-9ff8-f4f704a159d4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:02.970283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.970165    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f252503f-59b8-47c1-9ff8-f4f704a159d4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:02.970283 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:02.970190    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f252503f-59b8-47c1-9ff8-f4f704a159d4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.071224 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.071182    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f252503f-59b8-47c1-9ff8-f4f704a159d4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.071474 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.071236    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f252503f-59b8-47c1-9ff8-f4f704a159d4-web-config\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.071474 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.071274    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f252503f-59b8-47c1-9ff8-f4f704a159d4-config-out\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.071474 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.071296    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f252503f-59b8-47c1-9ff8-f4f704a159d4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.071474 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.071340    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f252503f-59b8-47c1-9ff8-f4f704a159d4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.071474 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.071371    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f252503f-59b8-47c1-9ff8-f4f704a159d4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.071474 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.071407    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f252503f-59b8-47c1-9ff8-f4f704a159d4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.071474 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.071437    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f252503f-59b8-47c1-9ff8-f4f704a159d4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.071474 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.071478    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f252503f-59b8-47c1-9ff8-f4f704a159d4-config-volume\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.071892 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.071499    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfw58\" (UniqueName: \"kubernetes.io/projected/f252503f-59b8-47c1-9ff8-f4f704a159d4-kube-api-access-pfw58\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.071892 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.071530    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f252503f-59b8-47c1-9ff8-f4f704a159d4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.071892 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.071574    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f252503f-59b8-47c1-9ff8-f4f704a159d4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.071892 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.071606    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f252503f-59b8-47c1-9ff8-f4f704a159d4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.071892 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.071693    5958 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:43:03.072146 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.072096    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f252503f-59b8-47c1-9ff8-f4f704a159d4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.074370 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.073613    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f252503f-59b8-47c1-9ff8-f4f704a159d4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.074370 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.073936    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f252503f-59b8-47c1-9ff8-f4f704a159d4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.075127 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.075068    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f252503f-59b8-47c1-9ff8-f4f704a159d4-config-out\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.075752 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.075707    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f252503f-59b8-47c1-9ff8-f4f704a159d4-config-volume\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.076971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.076921    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f252503f-59b8-47c1-9ff8-f4f704a159d4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.077119 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.077080    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f252503f-59b8-47c1-9ff8-f4f704a159d4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.077236 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.077140    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f252503f-59b8-47c1-9ff8-f4f704a159d4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.077662 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.077638    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f252503f-59b8-47c1-9ff8-f4f704a159d4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.077738 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.077650    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f252503f-59b8-47c1-9ff8-f4f704a159d4-web-config\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.078221 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.078189    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f252503f-59b8-47c1-9ff8-f4f704a159d4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.079842 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.079818    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f252503f-59b8-47c1-9ff8-f4f704a159d4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.134047 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.133974    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfw58\" (UniqueName: \"kubernetes.io/projected/f252503f-59b8-47c1-9ff8-f4f704a159d4-kube-api-access-pfw58\") pod \"alertmanager-main-0\" (UID: \"f252503f-59b8-47c1-9ff8-f4f704a159d4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.264967 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.264931    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:43:03.395760 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.395683    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d79cb0a2-da98-4c41-a201-50755411d3c1" path="/var/lib/kubelet/pods/d79cb0a2-da98-4c41-a201-50755411d3c1/volumes"
Oct 30 19:43:03.683942 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.683863    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-8698dc55b4-pwqbt"
Oct 30 19:43:03.781477 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:03.781417    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:43:03.784834 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:43:03.784798    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf252503f_59b8_47c1_9ff8_f4f704a159d4.slice/crio-b3188d0f268cb1b59d916d56ec18ee9040355cb4cfa4038b519516973ae26a4a WatchSource:0}: Error finding container b3188d0f268cb1b59d916d56ec18ee9040355cb4cfa4038b519516973ae26a4a: Status 404 returned error can't find the container with id b3188d0f268cb1b59d916d56ec18ee9040355cb4cfa4038b519516973ae26a4a
Oct 30 19:43:04.601174 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:04.601138    5958 generic.go:356] "Generic (PLEG): container finished" podID="f252503f-59b8-47c1-9ff8-f4f704a159d4" containerID="fa955c8e8a579243933611ab76e7dbf678c845c986510aeee24ad7c269ba8ddf" exitCode=0
Oct 30 19:43:04.601401 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:04.601235    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f252503f-59b8-47c1-9ff8-f4f704a159d4","Type":"ContainerDied","Data":"fa955c8e8a579243933611ab76e7dbf678c845c986510aeee24ad7c269ba8ddf"}
Oct 30 19:43:04.601401 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:04.601320    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f252503f-59b8-47c1-9ff8-f4f704a159d4","Type":"ContainerStarted","Data":"b3188d0f268cb1b59d916d56ec18ee9040355cb4cfa4038b519516973ae26a4a"}
Oct 30 19:43:04.871269 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:04.871238    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ft6pg"
Oct 30 19:43:05.609779 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:05.609740    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f252503f-59b8-47c1-9ff8-f4f704a159d4","Type":"ContainerStarted","Data":"767073fd5fe3e6f001c7ea0ba0120ead76cce4bc9eec080dce0a4a17c953976d"}
Oct 30 19:43:05.609985 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:05.609826    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f252503f-59b8-47c1-9ff8-f4f704a159d4","Type":"ContainerStarted","Data":"93acef886396cf5508ec0ddec649f3ebd653151ce05055403e5145181753a186"}
Oct 30 19:43:05.609985 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:05.609876    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f252503f-59b8-47c1-9ff8-f4f704a159d4","Type":"ContainerStarted","Data":"9b51ff638095042f8c300401f63d4d499b19a824e20394cc710d58bf6ba95941"}
Oct 30 19:43:05.609985 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:05.609915    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f252503f-59b8-47c1-9ff8-f4f704a159d4","Type":"ContainerStarted","Data":"998abcb6473a0a4e5cc01fc97780ed1861325acbe13de49d4187fbac2f305bba"}
Oct 30 19:43:05.609985 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:05.609941    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f252503f-59b8-47c1-9ff8-f4f704a159d4","Type":"ContainerStarted","Data":"314a0400eed2f0706b5d5e0b7e68c6d3efab32820ca6523d7f4a47537e14e7cd"}
Oct 30 19:43:05.609985 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:05.609975    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f252503f-59b8-47c1-9ff8-f4f704a159d4","Type":"ContainerStarted","Data":"52080ae72bfb7f4cbc2ccc74ff89ede89e946f476c92285310edd15e21e56bc3"}
Oct 30 19:43:05.740988 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:05.740919    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.729949981 podStartE2EDuration="3.740902441s" totalImagesPullingTime="0s" totalInitContainerRuntime="10.95246ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:05.73972822 +0000 UTC m=+24.471943612" watchObservedRunningTime="2026-10-30 19:43:05.740902441 +0000 UTC m=+24.473117840"
Oct 30 19:43:06.143664 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.143627    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:43:06.144147 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.143944    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="prometheus" containerID="cri-o://8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168" gracePeriod=600
Oct 30 19:43:06.144289 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.144252    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="kube-rbac-proxy-thanos" containerID="cri-o://7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2" gracePeriod=600
Oct 30 19:43:06.144379 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.144362    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="kube-rbac-proxy" containerID="cri-o://ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981" gracePeriod=600
Oct 30 19:43:06.144434 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.144421    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="kube-rbac-proxy-web" containerID="cri-o://b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf" gracePeriod=600
Oct 30 19:43:06.144504 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.144496    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="thanos-sidecar" containerID="cri-o://423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03" gracePeriod=600
Oct 30 19:43:06.144564 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.144552    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="config-reloader" containerID="cri-o://ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce" gracePeriod=600
Oct 30 19:43:06.469350 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.469312    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:06.611317 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.611284    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-1\") pod \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") "
Oct 30 19:43:06.611542 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.611337    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-metrics-client-ca\") pod \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") "
Oct 30 19:43:06.611542 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.611370    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-config-out\" (UniqueName: \"kubernetes.io/empty-dir/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-config-out\") pod \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") "
Oct 30 19:43:06.611542 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.611406    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-grpc-tls\") pod \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") "
Oct 30 19:43:06.611542 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.611438    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") "
Oct 30 19:43:06.611542 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.611486    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-0\") pod \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") "
Oct 30 19:43:06.611542 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.611516    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-kubelet-serving-ca-bundle\") pod \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") "
Oct 30 19:43:06.611858 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.611556    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-2\") pod \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") "
Oct 30 19:43:06.611858 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.611585    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-thanos-prometheus-http-client-file\") pod \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") "
Oct 30 19:43:06.611858 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.611618    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-kube-api-access-5kp8z\" (UniqueName: \"kubernetes.io/projected/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-kube-api-access-5kp8z\") pod \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") "
Oct 30 19:43:06.611858 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.611646    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-kube-rbac-proxy\") pod \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") "
Oct 30 19:43:06.611858 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.611672    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-tls-assets\" (UniqueName: \"kubernetes.io/projected/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-tls-assets\") pod \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") "
Oct 30 19:43:06.611858 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.611733    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-tls\") pod \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") "
Oct 30 19:43:06.611858 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.611761    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-metrics-client-certs\") pod \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") "
Oct 30 19:43:06.611858 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.611793    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-serving-certs-ca-bundle\") pod \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") "
Oct 30 19:43:06.611858 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.611821    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-web-config\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-web-config\") pod \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") "
Oct 30 19:43:06.611858 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.611846    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") "
Oct 30 19:43:06.612556 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.611885    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-config\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-config\") pod \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") "
Oct 30 19:43:06.612556 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.611983    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-db\") pod \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") "
Oct 30 19:43:06.612556 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.612017    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-trusted-ca-bundle\") pod \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\" (UID: \"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9\") "
Oct 30 19:43:06.612796 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.612736    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-1" pod "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" (UID: "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:06.612796 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.612777    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-trusted-ca-bundle" pod "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" (UID: "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:06.613540 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.613289    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-serving-certs-ca-bundle" pod "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" (UID: "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:06.614137 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.613858    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-metrics-client-ca" pod "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" (UID: "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:06.616267 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.616233    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-kubelet-serving-ca-bundle" pod "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" (UID: "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:06.616654 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.616630    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-2" pod "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" (UID: "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:06.616734 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.616662    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-db" pod "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" (UID: "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:06.617030 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.617000    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-0" pod "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" (UID: "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:06.618477 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.618318    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-kube-api-access-5kp8z" pod "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" (UID: "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9"). InnerVolumeSpecName "kube-api-access-5kp8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:06.629679 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:43:06.629643    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:43:06.637071 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.636737    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-tls-assets" pod "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" (UID: "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:06.637071 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.636880    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-config-out" pod "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" (UID: "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:06.637071 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.636949    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-tls" pod "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" (UID: "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:06.637375 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.637348    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-thanos-prometheus-http-client-file" pod "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" (UID: "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:06.637920 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.637604    5958 generic.go:356] "Generic (PLEG): container finished" podID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerID="7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2" exitCode=0
Oct 30 19:43:06.637920 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.637683    5958 generic.go:356] "Generic (PLEG): container finished" podID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerID="ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981" exitCode=0
Oct 30 19:43:06.637920 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.637692    5958 generic.go:356] "Generic (PLEG): container finished" podID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerID="b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf" exitCode=0
Oct 30 19:43:06.637920 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.637700    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:06.637920 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.637723    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9","Type":"ContainerDied","Data":"7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2"}
Oct 30 19:43:06.638240 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.637703    5958 generic.go:356] "Generic (PLEG): container finished" podID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerID="423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03" exitCode=0
Oct 30 19:43:06.638298 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.638238    5958 generic.go:356] "Generic (PLEG): container finished" podID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerID="ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce" exitCode=0
Oct 30 19:43:06.638298 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.638254    5958 generic.go:356] "Generic (PLEG): container finished" podID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerID="8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168" exitCode=0
Oct 30 19:43:06.638868 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.638486    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9","Type":"ContainerDied","Data":"ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981"}
Oct 30 19:43:06.638868 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.638545    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9","Type":"ContainerDied","Data":"b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf"}
Oct 30 19:43:06.638868 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.638588    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9","Type":"ContainerDied","Data":"423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03"}
Oct 30 19:43:06.638868 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.638626    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9","Type":"ContainerDied","Data":"ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce"}
Oct 30 19:43:06.638868 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.638665    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9","Type":"ContainerDied","Data":"8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168"}
Oct 30 19:43:06.638868 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.638704    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c2acb7dd-f543-4acd-91c5-aa4b48c2caf9","Type":"ContainerDied","Data":"8bdd2a1db11610307f5842206c66764c9cc590f10218b9fa345ffb3c8fffd446"}
Oct 30 19:43:06.638868 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.638752    5958 scope.go:122] "RemoveContainer" containerID="7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2"
Oct 30 19:43:06.638868 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.638760    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-grpc-tls" pod "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" (UID: "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:06.639298 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.638912    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-thanos-sidecar-tls" pod "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" (UID: "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:06.639298 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.638854    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-metrics-client-certs" pod "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" (UID: "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:06.639298 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.639025    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-kube-rbac-proxy" pod "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" (UID: "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:06.640958 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.640933    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-kube-rbac-proxy-web" pod "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" (UID: "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:06.641377 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.641339    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-config" pod "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" (UID: "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:06.650208 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.650109    5958 scope.go:122] "RemoveContainer" containerID="ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981"
Oct 30 19:43:06.652341 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.652314    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-web-config" pod "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" (UID: "c2acb7dd-f543-4acd-91c5-aa4b48c2caf9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:06.660238 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.660218    5958 scope.go:122] "RemoveContainer" containerID="b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf"
Oct 30 19:43:06.669599 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.669575    5958 scope.go:122] "RemoveContainer" containerID="423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03"
Oct 30 19:43:06.678797 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.678684    5958 scope.go:122] "RemoveContainer" containerID="ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce"
Oct 30 19:43:06.688144 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.688120    5958 scope.go:122] "RemoveContainer" containerID="8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168"
Oct 30 19:43:06.698672 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.698647    5958 scope.go:122] "RemoveContainer" containerID="83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef"
Oct 30 19:43:06.708102 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.708050    5958 scope.go:122] "RemoveContainer" containerID="7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2"
Oct 30 19:43:06.708414 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:43:06.708385    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2\": container with ID starting with 7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2 not found: ID does not exist" containerID="7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2"
Oct 30 19:43:06.708563 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.708425    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2"} err="failed to get container status \"7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2\": rpc error: code = NotFound desc = could not find container \"7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2\": container with ID starting with 7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2 not found: ID does not exist"
Oct 30 19:43:06.708563 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.708469    5958 scope.go:122] "RemoveContainer" containerID="ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981"
Oct 30 19:43:06.708824 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:43:06.708793    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981\": container with ID starting with ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981 not found: ID does not exist" containerID="ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981"
Oct 30 19:43:06.708939 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.708832    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981"} err="failed to get container status \"ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981\": rpc error: code = NotFound desc = could not find container \"ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981\": container with ID starting with ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981 not found: ID does not exist"
Oct 30 19:43:06.708939 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.708852    5958 scope.go:122] "RemoveContainer" containerID="b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf"
Oct 30 19:43:06.709290 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:43:06.709259    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf\": container with ID starting with b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf not found: ID does not exist" containerID="b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf"
Oct 30 19:43:06.709383 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.709294    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf"} err="failed to get container status \"b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf\": rpc error: code = NotFound desc = could not find container \"b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf\": container with ID starting with b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf not found: ID does not exist"
Oct 30 19:43:06.709383 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.709311    5958 scope.go:122] "RemoveContainer" containerID="423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03"
Oct 30 19:43:06.709758 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:43:06.709730    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03\": container with ID starting with 423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03 not found: ID does not exist" containerID="423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03"
Oct 30 19:43:06.709832 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.709766    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03"} err="failed to get container status \"423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03\": rpc error: code = NotFound desc = could not find container \"423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03\": container with ID starting with 423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03 not found: ID does not exist"
Oct 30 19:43:06.709832 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.709791    5958 scope.go:122] "RemoveContainer" containerID="ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce"
Oct 30 19:43:06.710098 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:43:06.710075    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce\": container with ID starting with ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce not found: ID does not exist" containerID="ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce"
Oct 30 19:43:06.710172 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.710101    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce"} err="failed to get container status \"ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce\": rpc error: code = NotFound desc = could not find container \"ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce\": container with ID starting with ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce not found: ID does not exist"
Oct 30 19:43:06.710172 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.710119    5958 scope.go:122] "RemoveContainer" containerID="8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168"
Oct 30 19:43:06.710405 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:43:06.710381    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168\": container with ID starting with 8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168 not found: ID does not exist" containerID="8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168"
Oct 30 19:43:06.710524 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.710410    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168"} err="failed to get container status \"8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168\": rpc error: code = NotFound desc = could not find container \"8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168\": container with ID starting with 8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168 not found: ID does not exist"
Oct 30 19:43:06.710524 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.710426    5958 scope.go:122] "RemoveContainer" containerID="83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef"
Oct 30 19:43:06.710769 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:43:06.710710    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef\": container with ID starting with 83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef not found: ID does not exist" containerID="83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef"
Oct 30 19:43:06.710769 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.710739    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef"} err="failed to get container status \"83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef\": rpc error: code = NotFound desc = could not find container \"83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef\": container with ID starting with 83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef not found: ID does not exist"
Oct 30 19:43:06.710769 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.710757    5958 scope.go:122] "RemoveContainer" containerID="7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2"
Oct 30 19:43:06.711148 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.711067    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2"} err="failed to get container status \"7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2\": rpc error: code = NotFound desc = could not find container \"7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2\": container with ID starting with 7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2 not found: ID does not exist"
Oct 30 19:43:06.711148 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.711094    5958 scope.go:122] "RemoveContainer" containerID="ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981"
Oct 30 19:43:06.711401 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.711363    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981"} err="failed to get container status \"ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981\": rpc error: code = NotFound desc = could not find container \"ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981\": container with ID starting with ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981 not found: ID does not exist"
Oct 30 19:43:06.711401 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.711385    5958 scope.go:122] "RemoveContainer" containerID="b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf"
Oct 30 19:43:06.711724 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.711701    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf"} err="failed to get container status \"b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf\": rpc error: code = NotFound desc = could not find container \"b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf\": container with ID starting with b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf not found: ID does not exist"
Oct 30 19:43:06.711801 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.711726    5958 scope.go:122] "RemoveContainer" containerID="423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03"
Oct 30 19:43:06.712032 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712010    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03"} err="failed to get container status \"423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03\": rpc error: code = NotFound desc = could not find container \"423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03\": container with ID starting with 423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03 not found: ID does not exist"
Oct 30 19:43:06.712032 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712032    5958 scope.go:122] "RemoveContainer" containerID="ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce"
Oct 30 19:43:06.712334 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712296    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce"} err="failed to get container status \"ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce\": rpc error: code = NotFound desc = could not find container \"ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce\": container with ID starting with ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce not found: ID does not exist"
Oct 30 19:43:06.712416 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712335    5958 scope.go:122] "RemoveContainer" containerID="8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168"
Oct 30 19:43:06.712637 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712613    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168"} err="failed to get container status \"8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168\": rpc error: code = NotFound desc = could not find container \"8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168\": container with ID starting with 8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168 not found: ID does not exist"
Oct 30 19:43:06.712702 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712640    5958 scope.go:122] "RemoveContainer" containerID="83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef"
Oct 30 19:43:06.712702 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712674    5958 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-kube-rbac-proxy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:06.712702 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712699    5958 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-tls-assets\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:06.712909 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712716    5958 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:06.712909 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712728    5958 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-metrics-client-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:06.712909 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712744    5958 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:06.712909 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712760    5958 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-web-config\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:06.712909 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712776    5958 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:06.712909 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712789    5958 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-config\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:06.712909 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712804    5958 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-db\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:06.712909 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712819    5958 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:06.712909 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712834    5958 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:06.712909 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712849    5958 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-metrics-client-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:06.712909 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712864    5958 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-config-out\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:06.712909 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712878    5958 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-grpc-tls\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:06.712909 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712893    5958 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:06.712909 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712908    5958 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:06.713545 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712921    5958 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:06.713545 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712915    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef"} err="failed to get container status \"83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef\": rpc error: code = NotFound desc = could not find container \"83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef\": container with ID starting with 83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef not found: ID does not exist"
Oct 30 19:43:06.713545 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712934    5958 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:06.713545 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712948    5958 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:06.713545 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712959    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kp8z\" (UniqueName: \"kubernetes.io/projected/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9-kube-api-access-5kp8z\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:43:06.713545 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.712938    5958 scope.go:122] "RemoveContainer" containerID="7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2"
Oct 30 19:43:06.713545 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.713226    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2"} err="failed to get container status \"7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2\": rpc error: code = NotFound desc = could not find container \"7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2\": container with ID starting with 7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2 not found: ID does not exist"
Oct 30 19:43:06.713545 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.713250    5958 scope.go:122] "RemoveContainer" containerID="ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981"
Oct 30 19:43:06.713912 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.713586    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981"} err="failed to get container status \"ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981\": rpc error: code = NotFound desc = could not find container \"ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981\": container with ID starting with ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981 not found: ID does not exist"
Oct 30 19:43:06.713912 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.713611    5958 scope.go:122] "RemoveContainer" containerID="b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf"
Oct 30 19:43:06.713912 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.713887    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf"} err="failed to get container status \"b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf\": rpc error: code = NotFound desc = could not find container \"b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf\": container with ID starting with b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf not found: ID does not exist"
Oct 30 19:43:06.713912 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.713910    5958 scope.go:122] "RemoveContainer" containerID="423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03"
Oct 30 19:43:06.714225 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.714198    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03"} err="failed to get container status \"423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03\": rpc error: code = NotFound desc = could not find container \"423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03\": container with ID starting with 423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03 not found: ID does not exist"
Oct 30 19:43:06.714225 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.714223    5958 scope.go:122] "RemoveContainer" containerID="ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce"
Oct 30 19:43:06.714536 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.714514    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce"} err="failed to get container status \"ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce\": rpc error: code = NotFound desc = could not find container \"ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce\": container with ID starting with ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce not found: ID does not exist"
Oct 30 19:43:06.714619 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.714537    5958 scope.go:122] "RemoveContainer" containerID="8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168"
Oct 30 19:43:06.714899 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.714806    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168"} err="failed to get container status \"8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168\": rpc error: code = NotFound desc = could not find container \"8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168\": container with ID starting with 8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168 not found: ID does not exist"
Oct 30 19:43:06.714899 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.714831    5958 scope.go:122] "RemoveContainer" containerID="83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef"
Oct 30 19:43:06.715112 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.715087    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef"} err="failed to get container status \"83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef\": rpc error: code = NotFound desc = could not find container \"83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef\": container with ID starting with 83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef not found: ID does not exist"
Oct 30 19:43:06.715201 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.715113    5958 scope.go:122] "RemoveContainer" containerID="7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2"
Oct 30 19:43:06.715426 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.715391    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2"} err="failed to get container status \"7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2\": rpc error: code = NotFound desc = could not find container \"7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2\": container with ID starting with 7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2 not found: ID does not exist"
Oct 30 19:43:06.715426 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.715424    5958 scope.go:122] "RemoveContainer" containerID="ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981"
Oct 30 19:43:06.715720 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.715700    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981"} err="failed to get container status \"ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981\": rpc error: code = NotFound desc = could not find container \"ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981\": container with ID starting with ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981 not found: ID does not exist"
Oct 30 19:43:06.715720 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.715719    5958 scope.go:122] "RemoveContainer" containerID="b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf"
Oct 30 19:43:06.716000 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.715975    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf"} err="failed to get container status \"b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf\": rpc error: code = NotFound desc = could not find container \"b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf\": container with ID starting with b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf not found: ID does not exist"
Oct 30 19:43:06.716095 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.716002    5958 scope.go:122] "RemoveContainer" containerID="423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03"
Oct 30 19:43:06.716319 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.716294    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03"} err="failed to get container status \"423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03\": rpc error: code = NotFound desc = could not find container \"423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03\": container with ID starting with 423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03 not found: ID does not exist"
Oct 30 19:43:06.716382 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.716322    5958 scope.go:122] "RemoveContainer" containerID="ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce"
Oct 30 19:43:06.716649 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.716624    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce"} err="failed to get container status \"ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce\": rpc error: code = NotFound desc = could not find container \"ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce\": container with ID starting with ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce not found: ID does not exist"
Oct 30 19:43:06.716899 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.716654    5958 scope.go:122] "RemoveContainer" containerID="8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168"
Oct 30 19:43:06.716969 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.716901    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168"} err="failed to get container status \"8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168\": rpc error: code = NotFound desc = could not find container \"8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168\": container with ID starting with 8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168 not found: ID does not exist"
Oct 30 19:43:06.716969 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.716916    5958 scope.go:122] "RemoveContainer" containerID="83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef"
Oct 30 19:43:06.717197 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.717173    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef"} err="failed to get container status \"83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef\": rpc error: code = NotFound desc = could not find container \"83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef\": container with ID starting with 83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef not found: ID does not exist"
Oct 30 19:43:06.717278 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.717197    5958 scope.go:122] "RemoveContainer" containerID="7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2"
Oct 30 19:43:06.717553 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.717510    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2"} err="failed to get container status \"7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2\": rpc error: code = NotFound desc = could not find container \"7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2\": container with ID starting with 7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2 not found: ID does not exist"
Oct 30 19:43:06.717553 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.717531    5958 scope.go:122] "RemoveContainer" containerID="ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981"
Oct 30 19:43:06.717807 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.717787    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981"} err="failed to get container status \"ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981\": rpc error: code = NotFound desc = could not find container \"ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981\": container with ID starting with ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981 not found: ID does not exist"
Oct 30 19:43:06.717859 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.717809    5958 scope.go:122] "RemoveContainer" containerID="b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf"
Oct 30 19:43:06.718137 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.718092    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf"} err="failed to get container status \"b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf\": rpc error: code = NotFound desc = could not find container \"b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf\": container with ID starting with b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf not found: ID does not exist"
Oct 30 19:43:06.718137 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.718116    5958 scope.go:122] "RemoveContainer" containerID="423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03"
Oct 30 19:43:06.718392 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.718373    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03"} err="failed to get container status \"423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03\": rpc error: code = NotFound desc = could not find container \"423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03\": container with ID starting with 423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03 not found: ID does not exist"
Oct 30 19:43:06.718464 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.718393    5958 scope.go:122] "RemoveContainer" containerID="ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce"
Oct 30 19:43:06.718850 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.718829    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce"} err="failed to get container status \"ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce\": rpc error: code = NotFound desc = could not find container \"ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce\": container with ID starting with ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce not found: ID does not exist"
Oct 30 19:43:06.718919 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.718851    5958 scope.go:122] "RemoveContainer" containerID="8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168"
Oct 30 19:43:06.719305 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.719283    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168"} err="failed to get container status \"8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168\": rpc error: code = NotFound desc = could not find container \"8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168\": container with ID starting with 8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168 not found: ID does not exist"
Oct 30 19:43:06.719401 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.719306    5958 scope.go:122] "RemoveContainer" containerID="83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef"
Oct 30 19:43:06.719624 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.719600    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef"} err="failed to get container status \"83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef\": rpc error: code = NotFound desc = could not find container \"83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef\": container with ID starting with 83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef not found: ID does not exist"
Oct 30 19:43:06.719624 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.719626    5958 scope.go:122] "RemoveContainer" containerID="7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2"
Oct 30 19:43:06.719972 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.719947    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2"} err="failed to get container status \"7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2\": rpc error: code = NotFound desc = could not find container \"7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2\": container with ID starting with 7eafe978cd0a53df18be9fdcc9b110a3473cf3774a044634f3c152c0cd427fb2 not found: ID does not exist"
Oct 30 19:43:06.720038 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.719974    5958 scope.go:122] "RemoveContainer" containerID="ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981"
Oct 30 19:43:06.720422 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.720391    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981"} err="failed to get container status \"ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981\": rpc error: code = NotFound desc = could not find container \"ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981\": container with ID starting with ce60475b8e33d14bde5f4384b2fe9f504909571a9bbb910818e29d26dcbcd981 not found: ID does not exist"
Oct 30 19:43:06.720422 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.720417    5958 scope.go:122] "RemoveContainer" containerID="b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf"
Oct 30 19:43:06.720749 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.720717    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf"} err="failed to get container status \"b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf\": rpc error: code = NotFound desc = could not find container \"b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf\": container with ID starting with b9173fef3f029fbd74f5ad667aa0db002c54adf8dabeaebc903b8b9e99add4cf not found: ID does not exist"
Oct 30 19:43:06.720749 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.720743    5958 scope.go:122] "RemoveContainer" containerID="423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03"
Oct 30 19:43:06.721054 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.721029    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03"} err="failed to get container status \"423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03\": rpc error: code = NotFound desc = could not find container \"423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03\": container with ID starting with 423d1b7be9021cd14604453f1129757be34212f103e6e9d94478c2c2f4f92f03 not found: ID does not exist"
Oct 30 19:43:06.721146 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.721057    5958 scope.go:122] "RemoveContainer" containerID="ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce"
Oct 30 19:43:06.721349 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.721314    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce"} err="failed to get container status \"ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce\": rpc error: code = NotFound desc = could not find container \"ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce\": container with ID starting with ae6023c7b88047c7d248397cca3cbd005ec3f739bacd0e64cace66b6ecc504ce not found: ID does not exist"
Oct 30 19:43:06.721349 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.721340    5958 scope.go:122] "RemoveContainer" containerID="8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168"
Oct 30 19:43:06.721661 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.721640    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168"} err="failed to get container status \"8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168\": rpc error: code = NotFound desc = could not find container \"8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168\": container with ID starting with 8e9a097889e22999153898e857934bb2487ea34443d75410003633c85ecc4168 not found: ID does not exist"
Oct 30 19:43:06.721721 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.721660    5958 scope.go:122] "RemoveContainer" containerID="83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef"
Oct 30 19:43:06.721935 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:06.721912    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef"} err="failed to get container status \"83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef\": rpc error: code = NotFound desc = could not find container \"83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef\": container with ID starting with 83a3f162ce147d290613e948788ffc6333ee2968982116a77ded995815c879ef not found: ID does not exist"
Oct 30 19:43:07.065948 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.065911    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:43:07.117398 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.115774    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:43:07.117398 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.116138    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:43:07.117398 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.116572    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="kube-rbac-proxy-web"
Oct 30 19:43:07.117398 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.116589    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="kube-rbac-proxy-web"
Oct 30 19:43:07.117398 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.116609    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="kube-rbac-proxy-thanos"
Oct 30 19:43:07.117398 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.116616    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="kube-rbac-proxy-thanos"
Oct 30 19:43:07.117398 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.116626    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="kube-rbac-proxy"
Oct 30 19:43:07.117398 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.116634    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="kube-rbac-proxy"
Oct 30 19:43:07.117398 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.116669    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="thanos-sidecar"
Oct 30 19:43:07.117398 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.116675    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="thanos-sidecar"
Oct 30 19:43:07.117398 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.116689    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="config-reloader"
Oct 30 19:43:07.117398 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.116695    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="config-reloader"
Oct 30 19:43:07.117398 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.116706    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="prometheus"
Oct 30 19:43:07.117398 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.116712    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="prometheus"
Oct 30 19:43:07.117398 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.116727    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="init-config-reloader"
Oct 30 19:43:07.117398 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.116734    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="init-config-reloader"
Oct 30 19:43:07.117398 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.116800    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="thanos-sidecar"
Oct 30 19:43:07.117398 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.116812    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="config-reloader"
Oct 30 19:43:07.117398 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.116820    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="prometheus"
Oct 30 19:43:07.117398 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.116830    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="init-config-reloader"
Oct 30 19:43:07.117398 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.116838    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="kube-rbac-proxy-web"
Oct 30 19:43:07.117398 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.116846    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="kube-rbac-proxy-thanos"
Oct 30 19:43:07.117398 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.116853    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" containerName="kube-rbac-proxy"
Oct 30 19:43:07.144698 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.144665    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.202287 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.202113    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 19:43:07.202287 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.202147    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 19:43:07.202287 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.202161    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 19:43:07.202287 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.202163    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 19:43:07.202287 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.202196    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 19:43:07.202287 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.202259    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 19:43:07.202762 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.202343    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 19:43:07.202762 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.202402    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d17vqncs64e3\""
Oct 30 19:43:07.202762 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.202499    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 19:43:07.202762 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.202602    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tcwj6\""
Oct 30 19:43:07.202762 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.202621    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 19:43:07.203415 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.203389    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 19:43:07.203935 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.203908    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 19:43:07.203935 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.203912    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 19:43:07.254249 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.254217    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 19:43:07.254484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.254280    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 19:43:07.255054 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.255026    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:43:07.317114 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.317084    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.317285 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.317124    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.317285 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.317146    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.317394 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.317276    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/041ef390-b8bb-43e8-9adf-6c074e1a13b0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.317394 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.317312    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.317394 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.317342    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/041ef390-b8bb-43e8-9adf-6c074e1a13b0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.317542 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.317400    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-web-config\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.317542 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.317435    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/041ef390-b8bb-43e8-9adf-6c074e1a13b0-config-out\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.317542 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.317483    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-config\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.317542 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.317538    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/041ef390-b8bb-43e8-9adf-6c074e1a13b0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.317723 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.317560    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.317723 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.317578    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/041ef390-b8bb-43e8-9adf-6c074e1a13b0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.317723 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.317624    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.317723 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.317653    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/041ef390-b8bb-43e8-9adf-6c074e1a13b0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.317723 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.317711    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/041ef390-b8bb-43e8-9adf-6c074e1a13b0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.317927 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.317745    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/041ef390-b8bb-43e8-9adf-6c074e1a13b0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.317927 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.317813    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/041ef390-b8bb-43e8-9adf-6c074e1a13b0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.317927 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.317844    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkwln\" (UniqueName: \"kubernetes.io/projected/041ef390-b8bb-43e8-9adf-6c074e1a13b0-kube-api-access-hkwln\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.317927 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.317889    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.318091 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.317958    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/041ef390-b8bb-43e8-9adf-6c074e1a13b0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.403263 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.403220    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2acb7dd-f543-4acd-91c5-aa4b48c2caf9" path="/var/lib/kubelet/pods/c2acb7dd-f543-4acd-91c5-aa4b48c2caf9/volumes"
Oct 30 19:43:07.419010 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.418963    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/041ef390-b8bb-43e8-9adf-6c074e1a13b0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.419188 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.419017    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.419188 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.419153    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/041ef390-b8bb-43e8-9adf-6c074e1a13b0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.419302 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.419198    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.419302 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.419225    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/041ef390-b8bb-43e8-9adf-6c074e1a13b0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.419302 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.419255    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/041ef390-b8bb-43e8-9adf-6c074e1a13b0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.419302 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.419279    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/041ef390-b8bb-43e8-9adf-6c074e1a13b0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.419581 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.419304    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/041ef390-b8bb-43e8-9adf-6c074e1a13b0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.419581 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.419331    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkwln\" (UniqueName: \"kubernetes.io/projected/041ef390-b8bb-43e8-9adf-6c074e1a13b0-kube-api-access-hkwln\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.419581 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.419357    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.419581 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.419415    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/041ef390-b8bb-43e8-9adf-6c074e1a13b0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.419581 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.419467    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.419581 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.419505    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.419581 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.419532    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.419581 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.419557    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/041ef390-b8bb-43e8-9adf-6c074e1a13b0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.419581 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.419581    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.420007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.419621    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/041ef390-b8bb-43e8-9adf-6c074e1a13b0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.420007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.419647    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-web-config\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.420007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.419672    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/041ef390-b8bb-43e8-9adf-6c074e1a13b0-config-out\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.420007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.419694    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-config\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.420007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.419855    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/041ef390-b8bb-43e8-9adf-6c074e1a13b0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.420441 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.420336    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/041ef390-b8bb-43e8-9adf-6c074e1a13b0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.420769 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.420741    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/041ef390-b8bb-43e8-9adf-6c074e1a13b0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.420888 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.420812    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/041ef390-b8bb-43e8-9adf-6c074e1a13b0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.422308 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.422284    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.423014 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.422994    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/041ef390-b8bb-43e8-9adf-6c074e1a13b0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.423498 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.423467    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/041ef390-b8bb-43e8-9adf-6c074e1a13b0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.423706 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.423685    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.423957 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.423937    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/041ef390-b8bb-43e8-9adf-6c074e1a13b0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.424224 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.424206    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/041ef390-b8bb-43e8-9adf-6c074e1a13b0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.424738 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.424716    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/041ef390-b8bb-43e8-9adf-6c074e1a13b0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.427708 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.427682    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-config\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.428957 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.428029    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/041ef390-b8bb-43e8-9adf-6c074e1a13b0-config-out\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.428957 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.428099    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-web-config\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.428957 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.428128    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.428957 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.428612    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.429231 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.429214    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.429279 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.429245    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.431192 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.431171    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/041ef390-b8bb-43e8-9adf-6c074e1a13b0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.482083 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.482004    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkwln\" (UniqueName: \"kubernetes.io/projected/041ef390-b8bb-43e8-9adf-6c074e1a13b0-kube-api-access-hkwln\") pod \"prometheus-k8s-0\" (UID: \"041ef390-b8bb-43e8-9adf-6c074e1a13b0\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:07.583043 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:07.583001    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:08.102417 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:08.102384    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:43:08.106971 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:43:08.106933    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041ef390_b8bb_43e8_9adf_6c074e1a13b0.slice/crio-eaf533527de4013c5d01d13c50232ddeedc4c3299a18f7b741e46fe1b981ff82 WatchSource:0}: Error finding container eaf533527de4013c5d01d13c50232ddeedc4c3299a18f7b741e46fe1b981ff82: Status 404 returned error can't find the container with id eaf533527de4013c5d01d13c50232ddeedc4c3299a18f7b741e46fe1b981ff82
Oct 30 19:43:08.650541 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:08.650503    5958 generic.go:356] "Generic (PLEG): container finished" podID="041ef390-b8bb-43e8-9adf-6c074e1a13b0" containerID="0b5467cbeca25abfb4f3d284baeb0a75b3f1899adfba7713fe25e72e66ad4170" exitCode=0
Oct 30 19:43:08.651015 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:08.650571    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"041ef390-b8bb-43e8-9adf-6c074e1a13b0","Type":"ContainerDied","Data":"0b5467cbeca25abfb4f3d284baeb0a75b3f1899adfba7713fe25e72e66ad4170"}
Oct 30 19:43:08.651015 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:08.650652    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"041ef390-b8bb-43e8-9adf-6c074e1a13b0","Type":"ContainerStarted","Data":"eaf533527de4013c5d01d13c50232ddeedc4c3299a18f7b741e46fe1b981ff82"}
Oct 30 19:43:09.658069 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:09.658036    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"041ef390-b8bb-43e8-9adf-6c074e1a13b0","Type":"ContainerStarted","Data":"14ce2c74cdca5c221ad3ba2f7d5f122cef6fc21e4940786a897c9dda4092abc7"}
Oct 30 19:43:09.658584 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:09.658096    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"041ef390-b8bb-43e8-9adf-6c074e1a13b0","Type":"ContainerStarted","Data":"aa761a15cd6b9decae267fbbfa6865464f6d804b296cd9c8ae665108adbcb514"}
Oct 30 19:43:09.658584 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:09.658120    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"041ef390-b8bb-43e8-9adf-6c074e1a13b0","Type":"ContainerStarted","Data":"b2225ee3926b2856c03e337594cdb61c766b1b90d4780785225224468769dcaa"}
Oct 30 19:43:09.658584 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:09.658145    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"041ef390-b8bb-43e8-9adf-6c074e1a13b0","Type":"ContainerStarted","Data":"a08c0b29d94e4dabf142fa0ebab7749e4de96ffa7324dcccdf7726d507734c1b"}
Oct 30 19:43:09.658584 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:09.658166    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"041ef390-b8bb-43e8-9adf-6c074e1a13b0","Type":"ContainerStarted","Data":"9c3f2e6cd467716a89f7670a85e46c762a7bd722439b49995102d31339542ec0"}
Oct 30 19:43:09.658584 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:09.658188    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"041ef390-b8bb-43e8-9adf-6c074e1a13b0","Type":"ContainerStarted","Data":"745f87358ab07f9d006cc68b5b5d804b669338c2ece7c965e9f4dbe15c9ccf22"}
Oct 30 19:43:09.812443 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:09.812360    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.796062316 podStartE2EDuration="2.812342578s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.280262ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:09.810875718 +0000 UTC m=+28.543091147" watchObservedRunningTime="2026-10-30 19:43:09.812342578 +0000 UTC m=+28.544557966"
Oct 30 19:43:12.583149 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:12.583112    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:17.136954 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:17.136925    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m9t6w"
Oct 30 19:43:36.604001 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:43:36.603948    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:43:40.388617 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:40.388575    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:40.404763 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:40.404736    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:43:40.785991 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:43:40.785919    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:44:06.604391 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:44:06.604346    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:44:36.604165 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:44:36.604115    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:45:06.604110 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:45:06.604071    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:45:36.604247 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:45:36.604207    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:46:06.603225 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:46:06.603144    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:46:36.603381 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:46:36.603346    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:47:06.604473 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:47:06.604419    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:47:36.605590 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:47:36.605550    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:47:41.423483 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:47:41.423440    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 19:47:41.426294 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:47:41.426271    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 19:48:06.604365 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:48:06.604323    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:48:16.665910 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:16.665879    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-xmvlg"]
Oct 30 19:48:16.667925 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:16.667910    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-xmvlg"
Oct 30 19:48:16.772691 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:16.772670    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 19:48:16.772815 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:16.772705    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:48:16.772929 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:16.772913    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-g8f5d\""
Oct 30 19:48:16.773469 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:16.773430    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-xmvlg"]
Oct 30 19:48:16.805076 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:16.805050    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7cll\" (UniqueName: \"kubernetes.io/projected/e2255b30-74e6-4426-9795-04c641edc110-kube-api-access-s7cll\") pod \"openshift-lws-operator-8465994787-xmvlg\" (UID: \"e2255b30-74e6-4426-9795-04c641edc110\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-xmvlg"
Oct 30 19:48:16.805180 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:16.805082    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2255b30-74e6-4426-9795-04c641edc110-tmp\") pod \"openshift-lws-operator-8465994787-xmvlg\" (UID: \"e2255b30-74e6-4426-9795-04c641edc110\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-xmvlg"
Oct 30 19:48:16.905343 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:16.905318    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7cll\" (UniqueName: \"kubernetes.io/projected/e2255b30-74e6-4426-9795-04c641edc110-kube-api-access-s7cll\") pod \"openshift-lws-operator-8465994787-xmvlg\" (UID: \"e2255b30-74e6-4426-9795-04c641edc110\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-xmvlg"
Oct 30 19:48:16.905434 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:16.905352    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2255b30-74e6-4426-9795-04c641edc110-tmp\") pod \"openshift-lws-operator-8465994787-xmvlg\" (UID: \"e2255b30-74e6-4426-9795-04c641edc110\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-xmvlg"
Oct 30 19:48:16.905777 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:16.905760    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2255b30-74e6-4426-9795-04c641edc110-tmp\") pod \"openshift-lws-operator-8465994787-xmvlg\" (UID: \"e2255b30-74e6-4426-9795-04c641edc110\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-xmvlg"
Oct 30 19:48:16.967737 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:16.967685    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7cll\" (UniqueName: \"kubernetes.io/projected/e2255b30-74e6-4426-9795-04c641edc110-kube-api-access-s7cll\") pod \"openshift-lws-operator-8465994787-xmvlg\" (UID: \"e2255b30-74e6-4426-9795-04c641edc110\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-xmvlg"
Oct 30 19:48:16.993537 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:16.993515    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-xmvlg"
Oct 30 19:48:17.371956 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:17.371926    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-xmvlg"]
Oct 30 19:48:17.375768 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:48:17.375743    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2255b30_74e6_4426_9795_04c641edc110.slice/crio-e39149228c306328e2782f1a458fa6005b55b83391f1053f2c4b8529f3544d69 WatchSource:0}: Error finding container e39149228c306328e2782f1a458fa6005b55b83391f1053f2c4b8529f3544d69: Status 404 returned error can't find the container with id e39149228c306328e2782f1a458fa6005b55b83391f1053f2c4b8529f3544d69
Oct 30 19:48:17.377287 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:17.377269    5958 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:48:17.735206 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:17.735125    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-xmvlg" event={"ID":"e2255b30-74e6-4426-9795-04c641edc110","Type":"ContainerStarted","Data":"e39149228c306328e2782f1a458fa6005b55b83391f1053f2c4b8529f3544d69"}
Oct 30 19:48:20.749089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:20.749056    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-xmvlg" event={"ID":"e2255b30-74e6-4426-9795-04c641edc110","Type":"ContainerStarted","Data":"5687eb3b6babf6b37e9fa092c3791c2458d25ca5fc963b3527ca13ff92d1bb2d"}
Oct 30 19:48:36.605113 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:48:36.605079    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:48:39.499634 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.499585    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-xmvlg" podStartSLOduration=20.356024235 podStartE2EDuration="23.499571946s" totalImagesPullingTime="3.143547711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:20.870769119 +0000 UTC m=+339.602984508" watchObservedRunningTime="2026-10-30 19:48:39.499571946 +0000 UTC m=+358.231787332"
Oct 30 19:48:39.500013 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.499955    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb"]
Oct 30 19:48:39.510093 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.510057    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb"
Oct 30 19:48:39.607177 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.607151    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16c731fa-8a91-4cd9-9f36-a3078026bbe3-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb\" (UID: \"16c731fa-8a91-4cd9-9f36-a3078026bbe3\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb"
Oct 30 19:48:39.607307 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.607184    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gtgb\" (UniqueName: \"kubernetes.io/projected/16c731fa-8a91-4cd9-9f36-a3078026bbe3-kube-api-access-5gtgb\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb\" (UID: \"16c731fa-8a91-4cd9-9f36-a3078026bbe3\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb"
Oct 30 19:48:39.607307 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.607209    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/16c731fa-8a91-4cd9-9f36-a3078026bbe3-manifests\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb\" (UID: \"16c731fa-8a91-4cd9-9f36-a3078026bbe3\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb"
Oct 30 19:48:39.607376 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.607346    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/16c731fa-8a91-4cd9-9f36-a3078026bbe3-metrics-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb\" (UID: \"16c731fa-8a91-4cd9-9f36-a3078026bbe3\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb"
Oct 30 19:48:39.607412 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.607373    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/16c731fa-8a91-4cd9-9f36-a3078026bbe3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb\" (UID: \"16c731fa-8a91-4cd9-9f36-a3078026bbe3\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb"
Oct 30 19:48:39.607412 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.607400    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16c731fa-8a91-4cd9-9f36-a3078026bbe3-tmp\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb\" (UID: \"16c731fa-8a91-4cd9-9f36-a3078026bbe3\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb"
Oct 30 19:48:39.623679 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.623653    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 19:48:39.623810 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.623723    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 19:48:39.623906 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.623890    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 19:48:39.623953 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.623942    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 19:48:39.623991 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.623946    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 19:48:39.623991 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.623977    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4vn6w\""
Oct 30 19:48:39.690654 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.690629    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb"]
Oct 30 19:48:39.708722 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.708702    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16c731fa-8a91-4cd9-9f36-a3078026bbe3-tmp\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb\" (UID: \"16c731fa-8a91-4cd9-9f36-a3078026bbe3\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb"
Oct 30 19:48:39.708815 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.708748    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16c731fa-8a91-4cd9-9f36-a3078026bbe3-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb\" (UID: \"16c731fa-8a91-4cd9-9f36-a3078026bbe3\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb"
Oct 30 19:48:39.708815 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.708767    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gtgb\" (UniqueName: \"kubernetes.io/projected/16c731fa-8a91-4cd9-9f36-a3078026bbe3-kube-api-access-5gtgb\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb\" (UID: \"16c731fa-8a91-4cd9-9f36-a3078026bbe3\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb"
Oct 30 19:48:39.708910 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.708885    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/16c731fa-8a91-4cd9-9f36-a3078026bbe3-manifests\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb\" (UID: \"16c731fa-8a91-4cd9-9f36-a3078026bbe3\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb"
Oct 30 19:48:39.709005 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.708988    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/16c731fa-8a91-4cd9-9f36-a3078026bbe3-metrics-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb\" (UID: \"16c731fa-8a91-4cd9-9f36-a3078026bbe3\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb"
Oct 30 19:48:39.709050 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.709018    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/16c731fa-8a91-4cd9-9f36-a3078026bbe3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb\" (UID: \"16c731fa-8a91-4cd9-9f36-a3078026bbe3\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb"
Oct 30 19:48:39.709277 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.709254    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/16c731fa-8a91-4cd9-9f36-a3078026bbe3-manifests\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb\" (UID: \"16c731fa-8a91-4cd9-9f36-a3078026bbe3\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb"
Oct 30 19:48:39.711233 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.711208    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16c731fa-8a91-4cd9-9f36-a3078026bbe3-tmp\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb\" (UID: \"16c731fa-8a91-4cd9-9f36-a3078026bbe3\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb"
Oct 30 19:48:39.711383 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.711362    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16c731fa-8a91-4cd9-9f36-a3078026bbe3-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb\" (UID: \"16c731fa-8a91-4cd9-9f36-a3078026bbe3\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb"
Oct 30 19:48:39.711424 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.711384    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/16c731fa-8a91-4cd9-9f36-a3078026bbe3-metrics-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb\" (UID: \"16c731fa-8a91-4cd9-9f36-a3078026bbe3\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb"
Oct 30 19:48:39.711492 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.711430    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/16c731fa-8a91-4cd9-9f36-a3078026bbe3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb\" (UID: \"16c731fa-8a91-4cd9-9f36-a3078026bbe3\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb"
Oct 30 19:48:39.809415 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.809353    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gtgb\" (UniqueName: \"kubernetes.io/projected/16c731fa-8a91-4cd9-9f36-a3078026bbe3-kube-api-access-5gtgb\") pod \"opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb\" (UID: \"16c731fa-8a91-4cd9-9f36-a3078026bbe3\") " pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb"
Oct 30 19:48:39.825206 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:39.825187    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb"
Oct 30 19:48:40.328341 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:40.328308    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb"]
Oct 30 19:48:40.331369 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:48:40.331344    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c731fa_8a91_4cd9_9f36_a3078026bbe3.slice/crio-ed71421675076699cef192466fddba6fccd418ddc8f1fd4cee8a662f74066170 WatchSource:0}: Error finding container ed71421675076699cef192466fddba6fccd418ddc8f1fd4cee8a662f74066170: Status 404 returned error can't find the container with id ed71421675076699cef192466fddba6fccd418ddc8f1fd4cee8a662f74066170
Oct 30 19:48:40.825252 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:40.825214    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb" event={"ID":"16c731fa-8a91-4cd9-9f36-a3078026bbe3","Type":"ContainerStarted","Data":"ed71421675076699cef192466fddba6fccd418ddc8f1fd4cee8a662f74066170"}
Oct 30 19:48:43.840899 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:43.840865    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb" event={"ID":"16c731fa-8a91-4cd9-9f36-a3078026bbe3","Type":"ContainerStarted","Data":"126b57b6f7a800187930d747212f0af385130e7c00938ec14131ae3881adc86f"}
Oct 30 19:48:44.846785 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:44.846750    5958 generic.go:356] "Generic (PLEG): container finished" podID="16c731fa-8a91-4cd9-9f36-a3078026bbe3" containerID="126b57b6f7a800187930d747212f0af385130e7c00938ec14131ae3881adc86f" exitCode=0
Oct 30 19:48:44.847130 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:44.846838    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb" event={"ID":"16c731fa-8a91-4cd9-9f36-a3078026bbe3","Type":"ContainerDied","Data":"126b57b6f7a800187930d747212f0af385130e7c00938ec14131ae3881adc86f"}
Oct 30 19:48:45.852275 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:45.852240    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb" event={"ID":"16c731fa-8a91-4cd9-9f36-a3078026bbe3","Type":"ContainerStarted","Data":"877e9e4075426a02669d5b36dda6298f824df1696324ceea45eed9875de38410"}
Oct 30 19:48:45.852724 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:45.852350    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb"
Oct 30 19:48:45.996973 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:45.996887    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb" podStartSLOduration=2.309113995 podStartE2EDuration="6.99687451s" totalImagesPullingTime="4.201370446s" totalInitContainerRuntime="486.390069ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:45.995029837 +0000 UTC m=+364.727245227" watchObservedRunningTime="2026-10-30 19:48:45.99687451 +0000 UTC m=+364.729089896"
Oct 30 19:48:56.859198 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:48:56.859122    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb"
Oct 30 19:49:00.710636 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:00.710600    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5794f57dfd-8mwbm"]
Oct 30 19:49:00.713955 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:00.713933    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-8mwbm"
Oct 30 19:49:00.767702 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:00.767676    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ef63ca8-92c2-4fe3-b6e3-02e709740cdf-cert\") pod \"lws-controller-manager-5794f57dfd-8mwbm\" (UID: \"7ef63ca8-92c2-4fe3-b6e3-02e709740cdf\") " pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-8mwbm"
Oct 30 19:49:00.767823 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:00.767713    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7ef63ca8-92c2-4fe3-b6e3-02e709740cdf-manager-config\") pod \"lws-controller-manager-5794f57dfd-8mwbm\" (UID: \"7ef63ca8-92c2-4fe3-b6e3-02e709740cdf\") " pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-8mwbm"
Oct 30 19:49:00.767823 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:00.767740    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ef63ca8-92c2-4fe3-b6e3-02e709740cdf-metrics-cert\") pod \"lws-controller-manager-5794f57dfd-8mwbm\" (UID: \"7ef63ca8-92c2-4fe3-b6e3-02e709740cdf\") " pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-8mwbm"
Oct 30 19:49:00.767933 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:00.767860    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjrqq\" (UniqueName: \"kubernetes.io/projected/7ef63ca8-92c2-4fe3-b6e3-02e709740cdf-kube-api-access-rjrqq\") pod \"lws-controller-manager-5794f57dfd-8mwbm\" (UID: \"7ef63ca8-92c2-4fe3-b6e3-02e709740cdf\") " pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-8mwbm"
Oct 30 19:49:00.771430 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:00.771400    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 30 19:49:00.819980 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:00.819958    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5794f57dfd-8mwbm"]
Oct 30 19:49:00.869135 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:00.869108    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7ef63ca8-92c2-4fe3-b6e3-02e709740cdf-manager-config\") pod \"lws-controller-manager-5794f57dfd-8mwbm\" (UID: \"7ef63ca8-92c2-4fe3-b6e3-02e709740cdf\") " pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-8mwbm"
Oct 30 19:49:00.869249 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:00.869144    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ef63ca8-92c2-4fe3-b6e3-02e709740cdf-metrics-cert\") pod \"lws-controller-manager-5794f57dfd-8mwbm\" (UID: \"7ef63ca8-92c2-4fe3-b6e3-02e709740cdf\") " pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-8mwbm"
Oct 30 19:49:00.869249 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:00.869168    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjrqq\" (UniqueName: \"kubernetes.io/projected/7ef63ca8-92c2-4fe3-b6e3-02e709740cdf-kube-api-access-rjrqq\") pod \"lws-controller-manager-5794f57dfd-8mwbm\" (UID: \"7ef63ca8-92c2-4fe3-b6e3-02e709740cdf\") " pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-8mwbm"
Oct 30 19:49:00.869249 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:00.869221    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ef63ca8-92c2-4fe3-b6e3-02e709740cdf-cert\") pod \"lws-controller-manager-5794f57dfd-8mwbm\" (UID: \"7ef63ca8-92c2-4fe3-b6e3-02e709740cdf\") " pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-8mwbm"
Oct 30 19:49:00.869733 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:00.869717    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7ef63ca8-92c2-4fe3-b6e3-02e709740cdf-manager-config\") pod \"lws-controller-manager-5794f57dfd-8mwbm\" (UID: \"7ef63ca8-92c2-4fe3-b6e3-02e709740cdf\") " pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-8mwbm"
Oct 30 19:49:00.885033 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:00.885006    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mzftv\""
Oct 30 19:49:00.885127 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:00.885109    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 30 19:49:00.885186 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:00.885141    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 30 19:49:00.892378 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:00.892353    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ef63ca8-92c2-4fe3-b6e3-02e709740cdf-cert\") pod \"lws-controller-manager-5794f57dfd-8mwbm\" (UID: \"7ef63ca8-92c2-4fe3-b6e3-02e709740cdf\") " pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-8mwbm"
Oct 30 19:49:00.892526 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:00.892506    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ef63ca8-92c2-4fe3-b6e3-02e709740cdf-metrics-cert\") pod \"lws-controller-manager-5794f57dfd-8mwbm\" (UID: \"7ef63ca8-92c2-4fe3-b6e3-02e709740cdf\") " pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-8mwbm"
Oct 30 19:49:00.937747 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:00.937726    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjrqq\" (UniqueName: \"kubernetes.io/projected/7ef63ca8-92c2-4fe3-b6e3-02e709740cdf-kube-api-access-rjrqq\") pod \"lws-controller-manager-5794f57dfd-8mwbm\" (UID: \"7ef63ca8-92c2-4fe3-b6e3-02e709740cdf\") " pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-8mwbm"
Oct 30 19:49:01.024753 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:01.024694    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-8mwbm"
Oct 30 19:49:01.402504 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:01.402476    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5794f57dfd-8mwbm"]
Oct 30 19:49:01.405530 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:49:01.405498    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef63ca8_92c2_4fe3_b6e3_02e709740cdf.slice/crio-f7ea8f1313a7639bea34790c4080c4450b56ab1d7bbf019e51acb55b8b6d45a6 WatchSource:0}: Error finding container f7ea8f1313a7639bea34790c4080c4450b56ab1d7bbf019e51acb55b8b6d45a6: Status 404 returned error can't find the container with id f7ea8f1313a7639bea34790c4080c4450b56ab1d7bbf019e51acb55b8b6d45a6
Oct 30 19:49:01.913970 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:01.913936    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-8mwbm" event={"ID":"7ef63ca8-92c2-4fe3-b6e3-02e709740cdf","Type":"ContainerStarted","Data":"f7ea8f1313a7639bea34790c4080c4450b56ab1d7bbf019e51acb55b8b6d45a6"}
Oct 30 19:49:03.029281 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:03.029247    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-64985d6c8c-7n6jf"]
Oct 30 19:49:03.034026 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:03.034003    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64985d6c8c-7n6jf"
Oct 30 19:49:03.096927 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:03.096888    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83edebaa-d6c0-401b-930c-a75f91343c70-tls-certs\") pod \"kube-auth-proxy-64985d6c8c-7n6jf\" (UID: \"83edebaa-d6c0-401b-930c-a75f91343c70\") " pod="openshift-ingress/kube-auth-proxy-64985d6c8c-7n6jf"
Oct 30 19:49:03.097050 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:03.096937    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83edebaa-d6c0-401b-930c-a75f91343c70-tmp\") pod \"kube-auth-proxy-64985d6c8c-7n6jf\" (UID: \"83edebaa-d6c0-401b-930c-a75f91343c70\") " pod="openshift-ingress/kube-auth-proxy-64985d6c8c-7n6jf"
Oct 30 19:49:03.097127 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:03.097101    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2975\" (UniqueName: \"kubernetes.io/projected/83edebaa-d6c0-401b-930c-a75f91343c70-kube-api-access-d2975\") pod \"kube-auth-proxy-64985d6c8c-7n6jf\" (UID: \"83edebaa-d6c0-401b-930c-a75f91343c70\") " pod="openshift-ingress/kube-auth-proxy-64985d6c8c-7n6jf"
Oct 30 19:49:03.139680 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:03.139650    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 19:49:03.139680 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:03.139663    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 19:49:03.139817 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:03.139719    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-4blrb\""
Oct 30 19:49:03.140439 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:03.140421    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64985d6c8c-7n6jf"]
Oct 30 19:49:03.189748 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:03.189729    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 19:49:03.189856 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:03.189749    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 19:49:03.198081 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:03.198058    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83edebaa-d6c0-401b-930c-a75f91343c70-tls-certs\") pod \"kube-auth-proxy-64985d6c8c-7n6jf\" (UID: \"83edebaa-d6c0-401b-930c-a75f91343c70\") " pod="openshift-ingress/kube-auth-proxy-64985d6c8c-7n6jf"
Oct 30 19:49:03.198176 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:03.198084    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83edebaa-d6c0-401b-930c-a75f91343c70-tmp\") pod \"kube-auth-proxy-64985d6c8c-7n6jf\" (UID: \"83edebaa-d6c0-401b-930c-a75f91343c70\") " pod="openshift-ingress/kube-auth-proxy-64985d6c8c-7n6jf"
Oct 30 19:49:03.198176 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:03.198124    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2975\" (UniqueName: \"kubernetes.io/projected/83edebaa-d6c0-401b-930c-a75f91343c70-kube-api-access-d2975\") pod \"kube-auth-proxy-64985d6c8c-7n6jf\" (UID: \"83edebaa-d6c0-401b-930c-a75f91343c70\") " pod="openshift-ingress/kube-auth-proxy-64985d6c8c-7n6jf"
Oct 30 19:49:03.200420 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:03.200395    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83edebaa-d6c0-401b-930c-a75f91343c70-tmp\") pod \"kube-auth-proxy-64985d6c8c-7n6jf\" (UID: \"83edebaa-d6c0-401b-930c-a75f91343c70\") " pod="openshift-ingress/kube-auth-proxy-64985d6c8c-7n6jf"
Oct 30 19:49:03.200536 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:03.200521    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83edebaa-d6c0-401b-930c-a75f91343c70-tls-certs\") pod \"kube-auth-proxy-64985d6c8c-7n6jf\" (UID: \"83edebaa-d6c0-401b-930c-a75f91343c70\") " pod="openshift-ingress/kube-auth-proxy-64985d6c8c-7n6jf"
Oct 30 19:49:03.261251 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:03.261226    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2975\" (UniqueName: \"kubernetes.io/projected/83edebaa-d6c0-401b-930c-a75f91343c70-kube-api-access-d2975\") pod \"kube-auth-proxy-64985d6c8c-7n6jf\" (UID: \"83edebaa-d6c0-401b-930c-a75f91343c70\") " pod="openshift-ingress/kube-auth-proxy-64985d6c8c-7n6jf"
Oct 30 19:49:03.346631 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:03.346608    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64985d6c8c-7n6jf"
Oct 30 19:49:03.718203 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:03.718113    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64985d6c8c-7n6jf"]
Oct 30 19:49:03.722508 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:49:03.722476    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83edebaa_d6c0_401b_930c_a75f91343c70.slice/crio-deea7807156800a25277c5bdd5a6b11ec9d242015e5fb7d43a0505deccac61ab WatchSource:0}: Error finding container deea7807156800a25277c5bdd5a6b11ec9d242015e5fb7d43a0505deccac61ab: Status 404 returned error can't find the container with id deea7807156800a25277c5bdd5a6b11ec9d242015e5fb7d43a0505deccac61ab
Oct 30 19:49:03.924331 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:03.924290    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64985d6c8c-7n6jf" event={"ID":"83edebaa-d6c0-401b-930c-a75f91343c70","Type":"ContainerStarted","Data":"deea7807156800a25277c5bdd5a6b11ec9d242015e5fb7d43a0505deccac61ab"}
Oct 30 19:49:04.930893 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:04.930850    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-8mwbm" event={"ID":"7ef63ca8-92c2-4fe3-b6e3-02e709740cdf","Type":"ContainerStarted","Data":"ab7388bedf86d8f675103a076a5690ce385f4255726b8cce8fba244b9ecc41ac"}
Oct 30 19:49:04.931290 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:04.930984    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-8mwbm"
Oct 30 19:49:05.055145 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:05.055088    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-8mwbm" podStartSLOduration=1.9742821149999998 podStartE2EDuration="5.055069185s" totalImagesPullingTime="3.08078707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:05.05343025 +0000 UTC m=+383.785645640" watchObservedRunningTime="2026-10-30 19:49:05.055069185 +0000 UTC m=+383.787284575"
Oct 30 19:49:06.605539 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:49:06.605500    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:49:06.940503 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:06.940423    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64985d6c8c-7n6jf" event={"ID":"83edebaa-d6c0-401b-930c-a75f91343c70","Type":"ContainerStarted","Data":"eb79160be947e9f1689637c31f00cb6ef24e4755276b7b8a404be1be64b73fe7"}
Oct 30 19:49:07.071595 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:07.071546    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-64985d6c8c-7n6jf" podStartSLOduration=2.148214201 podStartE2EDuration="5.071533142s" totalImagesPullingTime="2.923318941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:07.070403137 +0000 UTC m=+385.802618527" watchObservedRunningTime="2026-10-30 19:49:07.071533142 +0000 UTC m=+385.803748531"
Oct 30 19:49:15.937553 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:49:15.937524    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5794f57dfd-8mwbm"
Oct 30 19:49:36.604328 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:49:36.604284    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:50:06.604636 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:50:06.604586    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:50:32.486438 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:50:32.486408    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-gct5q"]
Oct 30 19:50:32.489355 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:50:32.489338    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-gct5q"
Oct 30 19:50:32.593048 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:50:32.593023    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 19:50:32.593190 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:50:32.593057    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 19:50:32.593190 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:50:32.593106    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-cv7gr\""
Oct 30 19:50:32.594950 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:50:32.594922    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-gct5q"]
Oct 30 19:50:32.611759 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:50:32.611668    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpfdv\" (UniqueName: \"kubernetes.io/projected/bc91bfd0-25ad-40c0-b87b-f6a792372cf8-kube-api-access-mpfdv\") pod \"odh-model-controller-f87fcf56c-gct5q\" (UID: \"bc91bfd0-25ad-40c0-b87b-f6a792372cf8\") " pod="opendatahub/odh-model-controller-f87fcf56c-gct5q"
Oct 30 19:50:32.611935 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:50:32.611763    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc91bfd0-25ad-40c0-b87b-f6a792372cf8-metrics-certs\") pod \"odh-model-controller-f87fcf56c-gct5q\" (UID: \"bc91bfd0-25ad-40c0-b87b-f6a792372cf8\") " pod="opendatahub/odh-model-controller-f87fcf56c-gct5q"
Oct 30 19:50:32.611935 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:50:32.611891    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc91bfd0-25ad-40c0-b87b-f6a792372cf8-cert\") pod \"odh-model-controller-f87fcf56c-gct5q\" (UID: \"bc91bfd0-25ad-40c0-b87b-f6a792372cf8\") " pod="opendatahub/odh-model-controller-f87fcf56c-gct5q"
Oct 30 19:50:32.712954 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:50:32.712928    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc91bfd0-25ad-40c0-b87b-f6a792372cf8-cert\") pod \"odh-model-controller-f87fcf56c-gct5q\" (UID: \"bc91bfd0-25ad-40c0-b87b-f6a792372cf8\") " pod="opendatahub/odh-model-controller-f87fcf56c-gct5q"
Oct 30 19:50:32.713076 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:50:32.712984    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpfdv\" (UniqueName: \"kubernetes.io/projected/bc91bfd0-25ad-40c0-b87b-f6a792372cf8-kube-api-access-mpfdv\") pod \"odh-model-controller-f87fcf56c-gct5q\" (UID: \"bc91bfd0-25ad-40c0-b87b-f6a792372cf8\") " pod="opendatahub/odh-model-controller-f87fcf56c-gct5q"
Oct 30 19:50:32.713076 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:50:32.713013    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc91bfd0-25ad-40c0-b87b-f6a792372cf8-metrics-certs\") pod \"odh-model-controller-f87fcf56c-gct5q\" (UID: \"bc91bfd0-25ad-40c0-b87b-f6a792372cf8\") " pod="opendatahub/odh-model-controller-f87fcf56c-gct5q"
Oct 30 19:50:32.715311 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:50:32.715291    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc91bfd0-25ad-40c0-b87b-f6a792372cf8-cert\") pod \"odh-model-controller-f87fcf56c-gct5q\" (UID: \"bc91bfd0-25ad-40c0-b87b-f6a792372cf8\") " pod="opendatahub/odh-model-controller-f87fcf56c-gct5q"
Oct 30 19:50:32.715395 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:50:32.715340    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc91bfd0-25ad-40c0-b87b-f6a792372cf8-metrics-certs\") pod \"odh-model-controller-f87fcf56c-gct5q\" (UID: \"bc91bfd0-25ad-40c0-b87b-f6a792372cf8\") " pod="opendatahub/odh-model-controller-f87fcf56c-gct5q"
Oct 30 19:50:32.774113 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:50:32.774064    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpfdv\" (UniqueName: \"kubernetes.io/projected/bc91bfd0-25ad-40c0-b87b-f6a792372cf8-kube-api-access-mpfdv\") pod \"odh-model-controller-f87fcf56c-gct5q\" (UID: \"bc91bfd0-25ad-40c0-b87b-f6a792372cf8\") " pod="opendatahub/odh-model-controller-f87fcf56c-gct5q"
Oct 30 19:50:32.801084 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:50:32.801059    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-gct5q"
Oct 30 19:50:33.172998 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:50:33.172969    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-gct5q"]
Oct 30 19:50:33.176038 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:50:33.176009    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc91bfd0_25ad_40c0_b87b_f6a792372cf8.slice/crio-92459a90bade52ee3aeece0059a9fcd9eeff6d76925faea5590a43728b1733a8 WatchSource:0}: Error finding container 92459a90bade52ee3aeece0059a9fcd9eeff6d76925faea5590a43728b1733a8: Status 404 returned error can't find the container with id 92459a90bade52ee3aeece0059a9fcd9eeff6d76925faea5590a43728b1733a8
Oct 30 19:50:33.284202 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:50:33.284167    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-gct5q" event={"ID":"bc91bfd0-25ad-40c0-b87b-f6a792372cf8","Type":"ContainerStarted","Data":"92459a90bade52ee3aeece0059a9fcd9eeff6d76925faea5590a43728b1733a8"}
Oct 30 19:50:36.302426 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:50:36.302346    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-gct5q" event={"ID":"bc91bfd0-25ad-40c0-b87b-f6a792372cf8","Type":"ContainerStarted","Data":"2add2f886d03d8571c91641ae030c2d8778e93f2968d564f65d332ea5fa63929"}
Oct 30 19:50:36.302426 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:50:36.302409    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-f87fcf56c-gct5q"
Oct 30 19:50:36.426793 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:50:36.426748    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-f87fcf56c-gct5q" podStartSLOduration=1.563107536 podStartE2EDuration="4.426734374s" totalImagesPullingTime="2.863626838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:36.425732384 +0000 UTC m=+475.157947772" watchObservedRunningTime="2026-10-30 19:50:36.426734374 +0000 UTC m=+475.158949763"
Oct 30 19:50:36.604038 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:50:36.604007    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:50:47.309597 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:50:47.309566    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-f87fcf56c-gct5q"
Oct 30 19:51:06.606710 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:51:06.606671    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:51:36.604786 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:51:36.604688    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:52:06.604854 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:52:06.604755    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:52:07.109387 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:52:07.109351    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-j2cqk"]
Oct 30 19:52:07.112878 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:52:07.112859    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-j2cqk"
Oct 30 19:52:07.178933 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:52:07.178902    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szp96\" (UniqueName: \"kubernetes.io/projected/0db352c1-b58a-4214-84ef-b507c534184a-kube-api-access-szp96\") pod \"authorino-operator-68d6dd76dc-j2cqk\" (UID: \"0db352c1-b58a-4214-84ef-b507c534184a\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-j2cqk"
Oct 30 19:52:07.229690 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:52:07.229655    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-l4rqv\""
Oct 30 19:52:07.229690 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:52:07.229679    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 19:52:07.229887 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:52:07.229657    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 19:52:07.230743 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:52:07.230725    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-j2cqk"]
Oct 30 19:52:07.279870 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:52:07.279839    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szp96\" (UniqueName: \"kubernetes.io/projected/0db352c1-b58a-4214-84ef-b507c534184a-kube-api-access-szp96\") pod \"authorino-operator-68d6dd76dc-j2cqk\" (UID: \"0db352c1-b58a-4214-84ef-b507c534184a\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-j2cqk"
Oct 30 19:52:07.346162 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:52:07.346137    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szp96\" (UniqueName: \"kubernetes.io/projected/0db352c1-b58a-4214-84ef-b507c534184a-kube-api-access-szp96\") pod \"authorino-operator-68d6dd76dc-j2cqk\" (UID: \"0db352c1-b58a-4214-84ef-b507c534184a\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-j2cqk"
Oct 30 19:52:07.424550 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:52:07.424476    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-j2cqk"
Oct 30 19:52:07.794259 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:52:07.794147    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-j2cqk"]
Oct 30 19:52:07.797055 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:52:07.797026    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db352c1_b58a_4214_84ef_b507c534184a.slice/crio-9ecc8cd11b107012649f6eafa1044e5bf75428b417879e230c6bef59c5e9cc6a WatchSource:0}: Error finding container 9ecc8cd11b107012649f6eafa1044e5bf75428b417879e230c6bef59c5e9cc6a: Status 404 returned error can't find the container with id 9ecc8cd11b107012649f6eafa1044e5bf75428b417879e230c6bef59c5e9cc6a
Oct 30 19:52:08.677021 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:52:08.676987    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-j2cqk" event={"ID":"0db352c1-b58a-4214-84ef-b507c534184a","Type":"ContainerStarted","Data":"9ecc8cd11b107012649f6eafa1044e5bf75428b417879e230c6bef59c5e9cc6a"}
Oct 30 19:52:18.718647 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:52:18.718621    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-j2cqk" event={"ID":"0db352c1-b58a-4214-84ef-b507c534184a","Type":"ContainerStarted","Data":"1f9ba45c24631eb6734de2587998e67c80c0b61fb009dbec193753137e472564"}
Oct 30 19:52:18.718947 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:52:18.718762    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-j2cqk"
Oct 30 19:52:18.878121 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:52:18.878056    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-j2cqk" podStartSLOduration=1.026646403 podStartE2EDuration="11.878044105s" totalImagesPullingTime="10.851397702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:18.876725243 +0000 UTC m=+577.608940632" watchObservedRunningTime="2026-10-30 19:52:18.878044105 +0000 UTC m=+577.610259494"
Oct 30 19:52:29.724502 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:52:29.724473    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-j2cqk"
Oct 30 19:52:36.603979 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:52:36.603946    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:52:41.459270 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:52:41.459236    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 19:52:41.460054 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:52:41.460028    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 19:53:06.603927 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:53:06.603888    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:53:17.870781 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:17.870741    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-f898w"]
Oct 30 19:53:17.874067 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:17.874051    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-f898w"
Oct 30 19:53:17.927115 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:17.927088    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5xpz2\""
Oct 30 19:53:17.954603 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:17.954568    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25hn8\" (UniqueName: \"kubernetes.io/projected/2acaebc2-2ff7-409a-b95b-44a8cc2491ee-kube-api-access-25hn8\") pod \"authorino-57f7f8659f-f898w\" (UID: \"2acaebc2-2ff7-409a-b95b-44a8cc2491ee\") " pod="kuadrant-system/authorino-57f7f8659f-f898w"
Oct 30 19:53:17.977577 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:17.977551    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-f898w"]
Oct 30 19:53:18.055857 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:18.055826    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25hn8\" (UniqueName: \"kubernetes.io/projected/2acaebc2-2ff7-409a-b95b-44a8cc2491ee-kube-api-access-25hn8\") pod \"authorino-57f7f8659f-f898w\" (UID: \"2acaebc2-2ff7-409a-b95b-44a8cc2491ee\") " pod="kuadrant-system/authorino-57f7f8659f-f898w"
Oct 30 19:53:18.117203 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:18.117171    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25hn8\" (UniqueName: \"kubernetes.io/projected/2acaebc2-2ff7-409a-b95b-44a8cc2491ee-kube-api-access-25hn8\") pod \"authorino-57f7f8659f-f898w\" (UID: \"2acaebc2-2ff7-409a-b95b-44a8cc2491ee\") " pod="kuadrant-system/authorino-57f7f8659f-f898w"
Oct 30 19:53:18.185242 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:18.185167    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-f898w"
Oct 30 19:53:18.553467 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:18.553377    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-f898w"]
Oct 30 19:53:18.556761 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:53:18.556735    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2acaebc2_2ff7_409a_b95b_44a8cc2491ee.slice/crio-cf0d2f84d995200539c65e2f76a5b8da0f8de49c957f0abb5e261974e6fc0bbc WatchSource:0}: Error finding container cf0d2f84d995200539c65e2f76a5b8da0f8de49c957f0abb5e261974e6fc0bbc: Status 404 returned error can't find the container with id cf0d2f84d995200539c65e2f76a5b8da0f8de49c957f0abb5e261974e6fc0bbc
Oct 30 19:53:18.558062 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:18.558044    5958 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:53:18.965039 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:18.965008    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-f898w" event={"ID":"2acaebc2-2ff7-409a-b95b-44a8cc2491ee","Type":"ContainerStarted","Data":"cf0d2f84d995200539c65e2f76a5b8da0f8de49c957f0abb5e261974e6fc0bbc"}
Oct 30 19:53:21.979401 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:21.979366    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-f898w" event={"ID":"2acaebc2-2ff7-409a-b95b-44a8cc2491ee","Type":"ContainerStarted","Data":"f03de89d6afdc7f4efd0c2c178fb2fb5648f14ffe3ebda30904964ac4ca1d0bb"}
Oct 30 19:53:25.149370 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:25.149302    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-f898w" podStartSLOduration=5.438623821 podStartE2EDuration="8.149282218s" totalImagesPullingTime="2.710658397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:22.096631452 +0000 UTC m=+640.828846842" watchObservedRunningTime="2026-10-30 19:53:25.149282218 +0000 UTC m=+643.881497608"
Oct 30 19:53:25.149952 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:25.149932    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-f898w"]
Oct 30 19:53:25.150098 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:25.150082    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-f898w" podUID="2acaebc2-2ff7-409a-b95b-44a8cc2491ee" containerName="authorino" containerID="cri-o://f03de89d6afdc7f4efd0c2c178fb2fb5648f14ffe3ebda30904964ac4ca1d0bb" gracePeriod=30
Oct 30 19:53:25.385862 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:25.385839    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-f898w"
Oct 30 19:53:25.527888 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:25.527810    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2acaebc2-2ff7-409a-b95b-44a8cc2491ee-kube-api-access-25hn8\" (UniqueName: \"kubernetes.io/projected/2acaebc2-2ff7-409a-b95b-44a8cc2491ee-kube-api-access-25hn8\") pod \"2acaebc2-2ff7-409a-b95b-44a8cc2491ee\" (UID: \"2acaebc2-2ff7-409a-b95b-44a8cc2491ee\") "
Oct 30 19:53:25.530020 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:25.529992    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2acaebc2-2ff7-409a-b95b-44a8cc2491ee-kube-api-access-25hn8" pod "2acaebc2-2ff7-409a-b95b-44a8cc2491ee" (UID: "2acaebc2-2ff7-409a-b95b-44a8cc2491ee"). InnerVolumeSpecName "kube-api-access-25hn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:25.629229 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:25.629188    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25hn8\" (UniqueName: \"kubernetes.io/projected/2acaebc2-2ff7-409a-b95b-44a8cc2491ee-kube-api-access-25hn8\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:26.002882 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:26.002838    5958 generic.go:356] "Generic (PLEG): container finished" podID="2acaebc2-2ff7-409a-b95b-44a8cc2491ee" containerID="f03de89d6afdc7f4efd0c2c178fb2fb5648f14ffe3ebda30904964ac4ca1d0bb" exitCode=0
Oct 30 19:53:26.002882 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:26.002875    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-f898w"
Oct 30 19:53:26.003148 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:26.002919    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-f898w" event={"ID":"2acaebc2-2ff7-409a-b95b-44a8cc2491ee","Type":"ContainerDied","Data":"f03de89d6afdc7f4efd0c2c178fb2fb5648f14ffe3ebda30904964ac4ca1d0bb"}
Oct 30 19:53:26.003148 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:26.002969    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-f898w" event={"ID":"2acaebc2-2ff7-409a-b95b-44a8cc2491ee","Type":"ContainerDied","Data":"cf0d2f84d995200539c65e2f76a5b8da0f8de49c957f0abb5e261974e6fc0bbc"}
Oct 30 19:53:26.003148 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:26.002990    5958 scope.go:122] "RemoveContainer" containerID="f03de89d6afdc7f4efd0c2c178fb2fb5648f14ffe3ebda30904964ac4ca1d0bb"
Oct 30 19:53:26.011657 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:26.011639    5958 scope.go:122] "RemoveContainer" containerID="f03de89d6afdc7f4efd0c2c178fb2fb5648f14ffe3ebda30904964ac4ca1d0bb"
Oct 30 19:53:26.011940 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:53:26.011922    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f03de89d6afdc7f4efd0c2c178fb2fb5648f14ffe3ebda30904964ac4ca1d0bb\": container with ID starting with f03de89d6afdc7f4efd0c2c178fb2fb5648f14ffe3ebda30904964ac4ca1d0bb not found: ID does not exist" containerID="f03de89d6afdc7f4efd0c2c178fb2fb5648f14ffe3ebda30904964ac4ca1d0bb"
Oct 30 19:53:26.011991 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:26.011949    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f03de89d6afdc7f4efd0c2c178fb2fb5648f14ffe3ebda30904964ac4ca1d0bb"} err="failed to get container status \"f03de89d6afdc7f4efd0c2c178fb2fb5648f14ffe3ebda30904964ac4ca1d0bb\": rpc error: code = NotFound desc = could not find container \"f03de89d6afdc7f4efd0c2c178fb2fb5648f14ffe3ebda30904964ac4ca1d0bb\": container with ID starting with f03de89d6afdc7f4efd0c2c178fb2fb5648f14ffe3ebda30904964ac4ca1d0bb not found: ID does not exist"
Oct 30 19:53:26.166565 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:26.166532    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-f898w"]
Oct 30 19:53:26.171672 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:26.171649    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-f898w"]
Oct 30 19:53:27.396397 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:27.396361    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2acaebc2-2ff7-409a-b95b-44a8cc2491ee" path="/var/lib/kubelet/pods/2acaebc2-2ff7-409a-b95b-44a8cc2491ee/volumes"
Oct 30 19:53:36.603954 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:53:36.603917    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:53:55.789039 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:55.789009    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-cwtjs"]
Oct 30 19:53:55.789778 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:55.789355    5958 cpu_manager.go:414] "containerMap: removing container" podUID="2acaebc2-2ff7-409a-b95b-44a8cc2491ee" containerName="authorino"
Oct 30 19:53:55.789778 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:55.789366    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2acaebc2-2ff7-409a-b95b-44a8cc2491ee" containerName="authorino"
Oct 30 19:53:55.789778 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:55.789431    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="2acaebc2-2ff7-409a-b95b-44a8cc2491ee" containerName="authorino"
Oct 30 19:53:55.792307 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:55.792288    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-cwtjs"
Oct 30 19:53:55.844935 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:55.844913    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5xpz2\""
Oct 30 19:53:55.873604 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:55.873574    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw4dq\" (UniqueName: \"kubernetes.io/projected/c69fb773-9d2b-45db-adee-c71f260a2253-kube-api-access-fw4dq\") pod \"authorino-6f6944bcdd-cwtjs\" (UID: \"c69fb773-9d2b-45db-adee-c71f260a2253\") " pod="kuadrant-system/authorino-6f6944bcdd-cwtjs"
Oct 30 19:53:55.895488 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:55.895465    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-cwtjs"]
Oct 30 19:53:55.974695 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:55.974668    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw4dq\" (UniqueName: \"kubernetes.io/projected/c69fb773-9d2b-45db-adee-c71f260a2253-kube-api-access-fw4dq\") pod \"authorino-6f6944bcdd-cwtjs\" (UID: \"c69fb773-9d2b-45db-adee-c71f260a2253\") " pod="kuadrant-system/authorino-6f6944bcdd-cwtjs"
Oct 30 19:53:56.036424 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:56.036396    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw4dq\" (UniqueName: \"kubernetes.io/projected/c69fb773-9d2b-45db-adee-c71f260a2253-kube-api-access-fw4dq\") pod \"authorino-6f6944bcdd-cwtjs\" (UID: \"c69fb773-9d2b-45db-adee-c71f260a2253\") " pod="kuadrant-system/authorino-6f6944bcdd-cwtjs"
Oct 30 19:53:56.103430 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:56.103408    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-cwtjs"
Oct 30 19:53:56.161318 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:56.161067    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-cwtjs"]
Oct 30 19:53:56.193141 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:56.193112    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f5d77845d-8srmg"]
Oct 30 19:53:56.196619 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:56.196604    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f5d77845d-8srmg"
Oct 30 19:53:56.253271 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:56.253250    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f5d77845d-8srmg"]
Oct 30 19:53:56.276796 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:56.276772    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99m8s\" (UniqueName: \"kubernetes.io/projected/f6789c17-74ae-4df2-9e0a-29a4286927a2-kube-api-access-99m8s\") pod \"authorino-5f5d77845d-8srmg\" (UID: \"f6789c17-74ae-4df2-9e0a-29a4286927a2\") " pod="kuadrant-system/authorino-5f5d77845d-8srmg"
Oct 30 19:53:56.377378 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:56.377321    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99m8s\" (UniqueName: \"kubernetes.io/projected/f6789c17-74ae-4df2-9e0a-29a4286927a2-kube-api-access-99m8s\") pod \"authorino-5f5d77845d-8srmg\" (UID: \"f6789c17-74ae-4df2-9e0a-29a4286927a2\") " pod="kuadrant-system/authorino-5f5d77845d-8srmg"
Oct 30 19:53:56.439236 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:56.439205    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99m8s\" (UniqueName: \"kubernetes.io/projected/f6789c17-74ae-4df2-9e0a-29a4286927a2-kube-api-access-99m8s\") pod \"authorino-5f5d77845d-8srmg\" (UID: \"f6789c17-74ae-4df2-9e0a-29a4286927a2\") " pod="kuadrant-system/authorino-5f5d77845d-8srmg"
Oct 30 19:53:56.470227 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:56.470205    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-cwtjs"]
Oct 30 19:53:56.473740 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:53:56.473717    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc69fb773_9d2b_45db_adee_c71f260a2253.slice/crio-9f5d75b206fbce15d9c96ae2cc856c7a354bfea0eef0b02549ffe862b65c4f20 WatchSource:0}: Error finding container 9f5d75b206fbce15d9c96ae2cc856c7a354bfea0eef0b02549ffe862b65c4f20: Status 404 returned error can't find the container with id 9f5d75b206fbce15d9c96ae2cc856c7a354bfea0eef0b02549ffe862b65c4f20
Oct 30 19:53:56.508645 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:56.508621    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f5d77845d-8srmg"
Oct 30 19:53:56.620197 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:56.620172    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f5d77845d-8srmg"]
Oct 30 19:53:56.649785 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:56.649732    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d4fcd4d-7nxzf"]
Oct 30 19:53:56.653050 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:56.653033    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d4fcd4d-7nxzf"
Oct 30 19:53:56.705673 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:56.705653    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 19:53:56.714753 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:56.714703    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d4fcd4d-7nxzf"]
Oct 30 19:53:56.780227 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:56.780201    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggtdr\" (UniqueName: \"kubernetes.io/projected/ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc-kube-api-access-ggtdr\") pod \"authorino-75d4fcd4d-7nxzf\" (UID: \"ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc\") " pod="kuadrant-system/authorino-75d4fcd4d-7nxzf"
Oct 30 19:53:56.780350 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:56.780249    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc-tls-cert\") pod \"authorino-75d4fcd4d-7nxzf\" (UID: \"ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc\") " pod="kuadrant-system/authorino-75d4fcd4d-7nxzf"
Oct 30 19:53:56.876267 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:56.876244    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f5d77845d-8srmg"]
Oct 30 19:53:56.879155 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:53:56.879136    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6789c17_74ae_4df2_9e0a_29a4286927a2.slice/crio-19cd51266fa16eb857e223cc8736cef3fb364b3396b0d4673cde685c79658751 WatchSource:0}: Error finding container 19cd51266fa16eb857e223cc8736cef3fb364b3396b0d4673cde685c79658751: Status 404 returned error can't find the container with id 19cd51266fa16eb857e223cc8736cef3fb364b3396b0d4673cde685c79658751
Oct 30 19:53:56.881203 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:56.881105    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggtdr\" (UniqueName: \"kubernetes.io/projected/ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc-kube-api-access-ggtdr\") pod \"authorino-75d4fcd4d-7nxzf\" (UID: \"ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc\") " pod="kuadrant-system/authorino-75d4fcd4d-7nxzf"
Oct 30 19:53:56.881203 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:56.881158    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc-tls-cert\") pod \"authorino-75d4fcd4d-7nxzf\" (UID: \"ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc\") " pod="kuadrant-system/authorino-75d4fcd4d-7nxzf"
Oct 30 19:53:56.883505 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:56.883489    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc-tls-cert\") pod \"authorino-75d4fcd4d-7nxzf\" (UID: \"ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc\") " pod="kuadrant-system/authorino-75d4fcd4d-7nxzf"
Oct 30 19:53:56.942982 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:56.942929    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggtdr\" (UniqueName: \"kubernetes.io/projected/ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc-kube-api-access-ggtdr\") pod \"authorino-75d4fcd4d-7nxzf\" (UID: \"ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc\") " pod="kuadrant-system/authorino-75d4fcd4d-7nxzf"
Oct 30 19:53:56.964878 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:56.964855    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d4fcd4d-7nxzf"
Oct 30 19:53:57.131436 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:57.131393    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f5d77845d-8srmg" event={"ID":"f6789c17-74ae-4df2-9e0a-29a4286927a2","Type":"ContainerStarted","Data":"19cd51266fa16eb857e223cc8736cef3fb364b3396b0d4673cde685c79658751"}
Oct 30 19:53:57.132652 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:57.132597    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-cwtjs" event={"ID":"c69fb773-9d2b-45db-adee-c71f260a2253","Type":"ContainerStarted","Data":"9f5d75b206fbce15d9c96ae2cc856c7a354bfea0eef0b02549ffe862b65c4f20"}
Oct 30 19:53:57.338891 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:57.338862    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d4fcd4d-7nxzf"]
Oct 30 19:53:57.342154 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:53:57.342115    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae195e92_06b3_49b5_a4d9_e6a9bd08ebdc.slice/crio-5f1134273045d89a7f673f222842d37d51d49f5270aa16fa4d28c4a2d32f5d21 WatchSource:0}: Error finding container 5f1134273045d89a7f673f222842d37d51d49f5270aa16fa4d28c4a2d32f5d21: Status 404 returned error can't find the container with id 5f1134273045d89a7f673f222842d37d51d49f5270aa16fa4d28c4a2d32f5d21
Oct 30 19:53:58.139070 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:58.139037    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-cwtjs" event={"ID":"c69fb773-9d2b-45db-adee-c71f260a2253","Type":"ContainerStarted","Data":"8c64bfeee10bdf9cdc045d117d47524a129f7afbd2c34c3ff9c1041bb3e75e17"}
Oct 30 19:53:58.139483 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:58.139077    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-cwtjs" podUID="c69fb773-9d2b-45db-adee-c71f260a2253" containerName="authorino" containerID="cri-o://8c64bfeee10bdf9cdc045d117d47524a129f7afbd2c34c3ff9c1041bb3e75e17" gracePeriod=30
Oct 30 19:53:58.140432 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:58.140411    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d4fcd4d-7nxzf" event={"ID":"ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc","Type":"ContainerStarted","Data":"5f1134273045d89a7f673f222842d37d51d49f5270aa16fa4d28c4a2d32f5d21"}
Oct 30 19:53:58.265884 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:58.265839    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-cwtjs" podStartSLOduration=2.104841138 podStartE2EDuration="3.265825358s" totalImagesPullingTime="1.16098422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:58.264516484 +0000 UTC m=+676.996731891" watchObservedRunningTime="2026-10-30 19:53:58.265825358 +0000 UTC m=+676.998040745"
Oct 30 19:53:58.385165 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:58.385140    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-cwtjs"
Oct 30 19:53:58.493980 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:58.493950    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c69fb773-9d2b-45db-adee-c71f260a2253-kube-api-access-fw4dq\" (UniqueName: \"kubernetes.io/projected/c69fb773-9d2b-45db-adee-c71f260a2253-kube-api-access-fw4dq\") pod \"c69fb773-9d2b-45db-adee-c71f260a2253\" (UID: \"c69fb773-9d2b-45db-adee-c71f260a2253\") "
Oct 30 19:53:58.496073 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:58.496049    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c69fb773-9d2b-45db-adee-c71f260a2253-kube-api-access-fw4dq" pod "c69fb773-9d2b-45db-adee-c71f260a2253" (UID: "c69fb773-9d2b-45db-adee-c71f260a2253"). InnerVolumeSpecName "kube-api-access-fw4dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:58.595463 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:58.595432    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw4dq\" (UniqueName: \"kubernetes.io/projected/c69fb773-9d2b-45db-adee-c71f260a2253-kube-api-access-fw4dq\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:53:59.145731 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:59.145700    5958 generic.go:356] "Generic (PLEG): container finished" podID="c69fb773-9d2b-45db-adee-c71f260a2253" containerID="8c64bfeee10bdf9cdc045d117d47524a129f7afbd2c34c3ff9c1041bb3e75e17" exitCode=0
Oct 30 19:53:59.146156 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:59.145745    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-cwtjs"
Oct 30 19:53:59.146156 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:59.145790    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-cwtjs" event={"ID":"c69fb773-9d2b-45db-adee-c71f260a2253","Type":"ContainerDied","Data":"8c64bfeee10bdf9cdc045d117d47524a129f7afbd2c34c3ff9c1041bb3e75e17"}
Oct 30 19:53:59.146156 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:59.145828    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-cwtjs" event={"ID":"c69fb773-9d2b-45db-adee-c71f260a2253","Type":"ContainerDied","Data":"9f5d75b206fbce15d9c96ae2cc856c7a354bfea0eef0b02549ffe862b65c4f20"}
Oct 30 19:53:59.146156 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:59.145852    5958 scope.go:122] "RemoveContainer" containerID="8c64bfeee10bdf9cdc045d117d47524a129f7afbd2c34c3ff9c1041bb3e75e17"
Oct 30 19:53:59.147417 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:59.147393    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f5d77845d-8srmg" event={"ID":"f6789c17-74ae-4df2-9e0a-29a4286927a2","Type":"ContainerStarted","Data":"4a6c9379c14c5523a2ae4189a111948ff51e354e722ecd3d603296cb32b35414"}
Oct 30 19:53:59.147655 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:59.147418    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5f5d77845d-8srmg" podUID="f6789c17-74ae-4df2-9e0a-29a4286927a2" containerName="authorino" containerID="cri-o://4a6c9379c14c5523a2ae4189a111948ff51e354e722ecd3d603296cb32b35414" gracePeriod=30
Oct 30 19:53:59.149389 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:59.149369    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d4fcd4d-7nxzf" event={"ID":"ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc","Type":"ContainerStarted","Data":"f4b2c5c7dc81d319b40177f2b229fe53b9b29f02bbd56aae69b8b69b2c3678f4"}
Oct 30 19:53:59.155787 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:59.155771    5958 scope.go:122] "RemoveContainer" containerID="8c64bfeee10bdf9cdc045d117d47524a129f7afbd2c34c3ff9c1041bb3e75e17"
Oct 30 19:53:59.156041 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:53:59.156021    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c64bfeee10bdf9cdc045d117d47524a129f7afbd2c34c3ff9c1041bb3e75e17\": container with ID starting with 8c64bfeee10bdf9cdc045d117d47524a129f7afbd2c34c3ff9c1041bb3e75e17 not found: ID does not exist" containerID="8c64bfeee10bdf9cdc045d117d47524a129f7afbd2c34c3ff9c1041bb3e75e17"
Oct 30 19:53:59.156090 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:59.156049    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c64bfeee10bdf9cdc045d117d47524a129f7afbd2c34c3ff9c1041bb3e75e17"} err="failed to get container status \"8c64bfeee10bdf9cdc045d117d47524a129f7afbd2c34c3ff9c1041bb3e75e17\": rpc error: code = NotFound desc = could not find container \"8c64bfeee10bdf9cdc045d117d47524a129f7afbd2c34c3ff9c1041bb3e75e17\": container with ID starting with 8c64bfeee10bdf9cdc045d117d47524a129f7afbd2c34c3ff9c1041bb3e75e17 not found: ID does not exist"
Oct 30 19:53:59.315865 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:59.315816    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-75d4fcd4d-7nxzf" podStartSLOduration=2.116284761 podStartE2EDuration="3.315799836s" totalImagesPullingTime="1.199515075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:59.314387002 +0000 UTC m=+678.046602391" watchObservedRunningTime="2026-10-30 19:53:59.315799836 +0000 UTC m=+678.048015225"
Oct 30 19:53:59.401784 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:59.401734    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f5d77845d-8srmg"
Oct 30 19:53:59.429609 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:59.429565    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5f5d77845d-8srmg" podStartSLOduration=2.212752433 podStartE2EDuration="3.429549294s" totalImagesPullingTime="1.216796861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:59.42908178 +0000 UTC m=+678.161297232" watchObservedRunningTime="2026-10-30 19:53:59.429549294 +0000 UTC m=+678.161764686"
Oct 30 19:53:59.501111 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:59.501085    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6789c17-74ae-4df2-9e0a-29a4286927a2-kube-api-access-99m8s\" (UniqueName: \"kubernetes.io/projected/f6789c17-74ae-4df2-9e0a-29a4286927a2-kube-api-access-99m8s\") pod \"f6789c17-74ae-4df2-9e0a-29a4286927a2\" (UID: \"f6789c17-74ae-4df2-9e0a-29a4286927a2\") "
Oct 30 19:53:59.503120 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:59.503090    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6789c17-74ae-4df2-9e0a-29a4286927a2-kube-api-access-99m8s" pod "f6789c17-74ae-4df2-9e0a-29a4286927a2" (UID: "f6789c17-74ae-4df2-9e0a-29a4286927a2"). InnerVolumeSpecName "kube-api-access-99m8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:59.540238 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:59.540215    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-cwtjs"]
Oct 30 19:53:59.544813 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:59.544792    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-cwtjs"]
Oct 30 19:53:59.602394 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:53:59.602365    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99m8s\" (UniqueName: \"kubernetes.io/projected/f6789c17-74ae-4df2-9e0a-29a4286927a2-kube-api-access-99m8s\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:54:00.155330 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:00.155300    5958 generic.go:356] "Generic (PLEG): container finished" podID="f6789c17-74ae-4df2-9e0a-29a4286927a2" containerID="4a6c9379c14c5523a2ae4189a111948ff51e354e722ecd3d603296cb32b35414" exitCode=0
Oct 30 19:54:00.155795 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:00.155353    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f5d77845d-8srmg"
Oct 30 19:54:00.155795 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:00.155395    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f5d77845d-8srmg" event={"ID":"f6789c17-74ae-4df2-9e0a-29a4286927a2","Type":"ContainerDied","Data":"4a6c9379c14c5523a2ae4189a111948ff51e354e722ecd3d603296cb32b35414"}
Oct 30 19:54:00.155795 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:00.155432    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f5d77845d-8srmg" event={"ID":"f6789c17-74ae-4df2-9e0a-29a4286927a2","Type":"ContainerDied","Data":"19cd51266fa16eb857e223cc8736cef3fb364b3396b0d4673cde685c79658751"}
Oct 30 19:54:00.155795 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:00.155476    5958 scope.go:122] "RemoveContainer" containerID="4a6c9379c14c5523a2ae4189a111948ff51e354e722ecd3d603296cb32b35414"
Oct 30 19:54:00.164552 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:00.164535    5958 scope.go:122] "RemoveContainer" containerID="4a6c9379c14c5523a2ae4189a111948ff51e354e722ecd3d603296cb32b35414"
Oct 30 19:54:00.164804 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:54:00.164787    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a6c9379c14c5523a2ae4189a111948ff51e354e722ecd3d603296cb32b35414\": container with ID starting with 4a6c9379c14c5523a2ae4189a111948ff51e354e722ecd3d603296cb32b35414 not found: ID does not exist" containerID="4a6c9379c14c5523a2ae4189a111948ff51e354e722ecd3d603296cb32b35414"
Oct 30 19:54:00.164849 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:00.164814    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a6c9379c14c5523a2ae4189a111948ff51e354e722ecd3d603296cb32b35414"} err="failed to get container status \"4a6c9379c14c5523a2ae4189a111948ff51e354e722ecd3d603296cb32b35414\": rpc error: code = NotFound desc = could not find container \"4a6c9379c14c5523a2ae4189a111948ff51e354e722ecd3d603296cb32b35414\": container with ID starting with 4a6c9379c14c5523a2ae4189a111948ff51e354e722ecd3d603296cb32b35414 not found: ID does not exist"
Oct 30 19:54:00.320879 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:00.320860    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f5d77845d-8srmg"]
Oct 30 19:54:00.325653 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:00.325631    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f5d77845d-8srmg"]
Oct 30 19:54:01.396485 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:01.396438    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c69fb773-9d2b-45db-adee-c71f260a2253" path="/var/lib/kubelet/pods/c69fb773-9d2b-45db-adee-c71f260a2253/volumes"
Oct 30 19:54:01.396823 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:01.396744    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6789c17-74ae-4df2-9e0a-29a4286927a2" path="/var/lib/kubelet/pods/f6789c17-74ae-4df2-9e0a-29a4286927a2/volumes"
Oct 30 19:54:06.604690 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:54:06.604655    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:54:16.319829 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:16.319789    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2fw7s"]
Oct 30 19:54:16.320214 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:16.320154    5958 cpu_manager.go:414] "containerMap: removing container" podUID="f6789c17-74ae-4df2-9e0a-29a4286927a2" containerName="authorino"
Oct 30 19:54:16.320214 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:16.320166    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6789c17-74ae-4df2-9e0a-29a4286927a2" containerName="authorino"
Oct 30 19:54:16.320214 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:16.320194    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c69fb773-9d2b-45db-adee-c71f260a2253" containerName="authorino"
Oct 30 19:54:16.320214 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:16.320203    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c69fb773-9d2b-45db-adee-c71f260a2253" containerName="authorino"
Oct 30 19:54:16.320356 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:16.320267    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c69fb773-9d2b-45db-adee-c71f260a2253" containerName="authorino"
Oct 30 19:54:16.320356 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:16.320281    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6789c17-74ae-4df2-9e0a-29a4286927a2" containerName="authorino"
Oct 30 19:54:16.324475 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:16.324441    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2fw7s"
Oct 30 19:54:16.379394 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:16.379369    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 30 19:54:16.379541 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:16.379436    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 30 19:54:16.381103 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:16.381088    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 30 19:54:16.429379 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:16.429350    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-2mvws\""
Oct 30 19:54:16.430347 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:16.430328    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2fw7s"]
Oct 30 19:54:16.430659 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:16.430637    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2543c3b9-ba23-4040-adbb-048caa38ca10-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-2fw7s\" (UID: \"2543c3b9-ba23-4040-adbb-048caa38ca10\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2fw7s"
Oct 30 19:54:16.430815 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:16.430795    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2543c3b9-ba23-4040-adbb-048caa38ca10-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-2fw7s\" (UID: \"2543c3b9-ba23-4040-adbb-048caa38ca10\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2fw7s"
Oct 30 19:54:16.430851 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:16.430842    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2543c3b9-ba23-4040-adbb-048caa38ca10-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-2fw7s\" (UID: \"2543c3b9-ba23-4040-adbb-048caa38ca10\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2fw7s"
Oct 30 19:54:16.430926 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:16.430909    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpktv\" (UniqueName: \"kubernetes.io/projected/2543c3b9-ba23-4040-adbb-048caa38ca10-kube-api-access-qpktv\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-2fw7s\" (UID: \"2543c3b9-ba23-4040-adbb-048caa38ca10\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2fw7s"
Oct 30 19:54:16.531624 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:16.531594    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2543c3b9-ba23-4040-adbb-048caa38ca10-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-2fw7s\" (UID: \"2543c3b9-ba23-4040-adbb-048caa38ca10\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2fw7s"
Oct 30 19:54:16.531760 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:16.531651    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2543c3b9-ba23-4040-adbb-048caa38ca10-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-2fw7s\" (UID: \"2543c3b9-ba23-4040-adbb-048caa38ca10\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2fw7s"
Oct 30 19:54:16.531760 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:16.531683    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpktv\" (UniqueName: \"kubernetes.io/projected/2543c3b9-ba23-4040-adbb-048caa38ca10-kube-api-access-qpktv\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-2fw7s\" (UID: \"2543c3b9-ba23-4040-adbb-048caa38ca10\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2fw7s"
Oct 30 19:54:16.531760 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:16.531727    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2543c3b9-ba23-4040-adbb-048caa38ca10-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-2fw7s\" (UID: \"2543c3b9-ba23-4040-adbb-048caa38ca10\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2fw7s"
Oct 30 19:54:16.531934 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:54:16.531916    5958 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Oct 30 19:54:16.532016 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:54:16.532005    5958 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2543c3b9-ba23-4040-adbb-048caa38ca10-proxy-source podName:2543c3b9-ba23-4040-adbb-048caa38ca10 nodeName:}" failed. No retries permitted until 2026-10-30 19:54:17.031985192 +0000 UTC m=+695.764200574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/2543c3b9-ba23-4040-adbb-048caa38ca10-proxy-source") pod "usage-logs-tenancy-proxy-7d94f55f4b-2fw7s" (UID: "2543c3b9-ba23-4040-adbb-048caa38ca10") : configmap "usage-tenancy-proxy-source" not found
Oct 30 19:54:16.532474 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:16.532427    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2543c3b9-ba23-4040-adbb-048caa38ca10-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-2fw7s\" (UID: \"2543c3b9-ba23-4040-adbb-048caa38ca10\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2fw7s"
Oct 30 19:54:16.535014 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:16.534991    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2543c3b9-ba23-4040-adbb-048caa38ca10-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-2fw7s\" (UID: \"2543c3b9-ba23-4040-adbb-048caa38ca10\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2fw7s"
Oct 30 19:54:16.594710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:16.594658    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpktv\" (UniqueName: \"kubernetes.io/projected/2543c3b9-ba23-4040-adbb-048caa38ca10-kube-api-access-qpktv\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-2fw7s\" (UID: \"2543c3b9-ba23-4040-adbb-048caa38ca10\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2fw7s"
Oct 30 19:54:17.035511 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:17.035432    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2543c3b9-ba23-4040-adbb-048caa38ca10-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-2fw7s\" (UID: \"2543c3b9-ba23-4040-adbb-048caa38ca10\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2fw7s"
Oct 30 19:54:17.036109 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:17.036091    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2543c3b9-ba23-4040-adbb-048caa38ca10-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-2fw7s\" (UID: \"2543c3b9-ba23-4040-adbb-048caa38ca10\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2fw7s"
Oct 30 19:54:17.236262 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:17.236238    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2fw7s"
Oct 30 19:54:17.623789 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:17.623760    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2fw7s"]
Oct 30 19:54:17.626354 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:54:17.626329    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2543c3b9_ba23_4040_adbb_048caa38ca10.slice/crio-3e4a82b42b83c4d96dc87b21ed37b0fc1e985389f0ac9c52a026830b339b2a11 WatchSource:0}: Error finding container 3e4a82b42b83c4d96dc87b21ed37b0fc1e985389f0ac9c52a026830b339b2a11: Status 404 returned error can't find the container with id 3e4a82b42b83c4d96dc87b21ed37b0fc1e985389f0ac9c52a026830b339b2a11
Oct 30 19:54:18.234962 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:18.234933    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2fw7s" event={"ID":"2543c3b9-ba23-4040-adbb-048caa38ca10","Type":"ContainerStarted","Data":"3e4a82b42b83c4d96dc87b21ed37b0fc1e985389f0ac9c52a026830b339b2a11"}
Oct 30 19:54:30.300680 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:30.300647    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2fw7s" event={"ID":"2543c3b9-ba23-4040-adbb-048caa38ca10","Type":"ContainerStarted","Data":"4e8704cc645d3ff69c6c286cbccdfd025dfd65414e0bce1efc146f3dd3287262"}
Oct 30 19:54:30.301061 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:30.300765    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2fw7s"
Oct 30 19:54:30.421659 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:30.421607    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2fw7s" podStartSLOduration=2.361982915 podStartE2EDuration="14.421590959s" totalImagesPullingTime="12.059608044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:30.420181599 +0000 UTC m=+709.152396989" watchObservedRunningTime="2026-10-30 19:54:30.421590959 +0000 UTC m=+709.153806349"
Oct 30 19:54:36.604624 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:54:36.604586    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:54:41.309858 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:54:41.309828    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-2fw7s"
Oct 30 19:55:06.603945 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:55:06.603869    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:55:36.604166 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:55:36.604132    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:55:48.361184 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.361153    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"]
Oct 30 19:55:48.363502 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.363484    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"
Oct 30 19:55:48.421790 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.421760    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 19:55:48.421925 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.421846    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 19:55:48.471787 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.471761    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lfm4p\""
Oct 30 19:55:48.471936 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.471830    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 19:55:48.472937 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.472920    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"]
Oct 30 19:55:48.526694 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.526673    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0df903f0-1e53-4243-b2d9-77f470486d7e-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh\" (UID: \"0df903f0-1e53-4243-b2d9-77f470486d7e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"
Oct 30 19:55:48.526816 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.526714    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkjnp\" (UniqueName: \"kubernetes.io/projected/0df903f0-1e53-4243-b2d9-77f470486d7e-kube-api-access-jkjnp\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh\" (UID: \"0df903f0-1e53-4243-b2d9-77f470486d7e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"
Oct 30 19:55:48.526957 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.526937    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0df903f0-1e53-4243-b2d9-77f470486d7e-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh\" (UID: \"0df903f0-1e53-4243-b2d9-77f470486d7e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"
Oct 30 19:55:48.527069 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.526973    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0df903f0-1e53-4243-b2d9-77f470486d7e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh\" (UID: \"0df903f0-1e53-4243-b2d9-77f470486d7e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"
Oct 30 19:55:48.527113 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.527092    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0df903f0-1e53-4243-b2d9-77f470486d7e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh\" (UID: \"0df903f0-1e53-4243-b2d9-77f470486d7e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"
Oct 30 19:55:48.527147 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.527121    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0df903f0-1e53-4243-b2d9-77f470486d7e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh\" (UID: \"0df903f0-1e53-4243-b2d9-77f470486d7e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"
Oct 30 19:55:48.627812 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.627752    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkjnp\" (UniqueName: \"kubernetes.io/projected/0df903f0-1e53-4243-b2d9-77f470486d7e-kube-api-access-jkjnp\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh\" (UID: \"0df903f0-1e53-4243-b2d9-77f470486d7e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"
Oct 30 19:55:48.627812 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.627793    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0df903f0-1e53-4243-b2d9-77f470486d7e-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh\" (UID: \"0df903f0-1e53-4243-b2d9-77f470486d7e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"
Oct 30 19:55:48.627972 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.627815    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0df903f0-1e53-4243-b2d9-77f470486d7e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh\" (UID: \"0df903f0-1e53-4243-b2d9-77f470486d7e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"
Oct 30 19:55:48.627972 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.627838    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0df903f0-1e53-4243-b2d9-77f470486d7e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh\" (UID: \"0df903f0-1e53-4243-b2d9-77f470486d7e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"
Oct 30 19:55:48.627972 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.627852    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0df903f0-1e53-4243-b2d9-77f470486d7e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh\" (UID: \"0df903f0-1e53-4243-b2d9-77f470486d7e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"
Oct 30 19:55:48.627972 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.627889    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0df903f0-1e53-4243-b2d9-77f470486d7e-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh\" (UID: \"0df903f0-1e53-4243-b2d9-77f470486d7e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"
Oct 30 19:55:48.628243 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.628216    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0df903f0-1e53-4243-b2d9-77f470486d7e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh\" (UID: \"0df903f0-1e53-4243-b2d9-77f470486d7e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"
Oct 30 19:55:48.628354 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.628268    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0df903f0-1e53-4243-b2d9-77f470486d7e-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh\" (UID: \"0df903f0-1e53-4243-b2d9-77f470486d7e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"
Oct 30 19:55:48.628354 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.628285    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0df903f0-1e53-4243-b2d9-77f470486d7e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh\" (UID: \"0df903f0-1e53-4243-b2d9-77f470486d7e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"
Oct 30 19:55:48.630166 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.630146    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0df903f0-1e53-4243-b2d9-77f470486d7e-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh\" (UID: \"0df903f0-1e53-4243-b2d9-77f470486d7e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"
Oct 30 19:55:48.630253 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.630239    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0df903f0-1e53-4243-b2d9-77f470486d7e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh\" (UID: \"0df903f0-1e53-4243-b2d9-77f470486d7e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"
Oct 30 19:55:48.702799 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.702767    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkjnp\" (UniqueName: \"kubernetes.io/projected/0df903f0-1e53-4243-b2d9-77f470486d7e-kube-api-access-jkjnp\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh\" (UID: \"0df903f0-1e53-4243-b2d9-77f470486d7e\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"
Oct 30 19:55:48.974926 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:48.974863    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"
Oct 30 19:55:49.347924 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:49.347838    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"]
Oct 30 19:55:49.350528 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:55:49.350502    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df903f0_1e53_4243_b2d9_77f470486d7e.slice/crio-4905590129faffc2f8fdaea02400e9ae96a69d50d02218357e07a8fc9d8c40f4 WatchSource:0}: Error finding container 4905590129faffc2f8fdaea02400e9ae96a69d50d02218357e07a8fc9d8c40f4: Status 404 returned error can't find the container with id 4905590129faffc2f8fdaea02400e9ae96a69d50d02218357e07a8fc9d8c40f4
Oct 30 19:55:49.627298 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:49.627242    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh" event={"ID":"0df903f0-1e53-4243-b2d9-77f470486d7e","Type":"ContainerStarted","Data":"4905590129faffc2f8fdaea02400e9ae96a69d50d02218357e07a8fc9d8c40f4"}
Oct 30 19:55:52.643738 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:52.643701    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh" event={"ID":"0df903f0-1e53-4243-b2d9-77f470486d7e","Type":"ContainerStarted","Data":"177de53df81575da2c3fe9c1c504f06214979d83bf15f929c9410836324d9679"}
Oct 30 19:55:52.768107 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:52.768056    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh" podStartSLOduration=2.3533237590000002 podStartE2EDuration="4.768042054s" totalImagesPullingTime="2.414718295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:52.764881554 +0000 UTC m=+791.497096944" watchObservedRunningTime="2026-10-30 19:55:52.768042054 +0000 UTC m=+791.500257444"
Oct 30 19:55:52.975413 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:52.975339    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"
Oct 30 19:55:58.975407 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:58.975374    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"
Oct 30 19:55:58.987553 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:58.987528    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"
Oct 30 19:55:59.492405 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:59.492377    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"]
Oct 30 19:55:59.495721 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:59.495704    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"
Oct 30 19:55:59.550337 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:59.550306    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 19:55:59.601208 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:59.601172    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"]
Oct 30 19:55:59.621152 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:59.621121    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f55f875c-cde6-43b0-bd95-ead043376d8f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295\" (UID: \"f55f875c-cde6-43b0-bd95-ead043376d8f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"
Oct 30 19:55:59.621275 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:59.621161    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f55f875c-cde6-43b0-bd95-ead043376d8f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295\" (UID: \"f55f875c-cde6-43b0-bd95-ead043376d8f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"
Oct 30 19:55:59.621275 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:59.621191    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f55f875c-cde6-43b0-bd95-ead043376d8f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295\" (UID: \"f55f875c-cde6-43b0-bd95-ead043376d8f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"
Oct 30 19:55:59.621353 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:59.621287    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6b79\" (UniqueName: \"kubernetes.io/projected/f55f875c-cde6-43b0-bd95-ead043376d8f-kube-api-access-j6b79\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295\" (UID: \"f55f875c-cde6-43b0-bd95-ead043376d8f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"
Oct 30 19:55:59.621353 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:59.621319    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f55f875c-cde6-43b0-bd95-ead043376d8f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295\" (UID: \"f55f875c-cde6-43b0-bd95-ead043376d8f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"
Oct 30 19:55:59.621353 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:59.621340    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f55f875c-cde6-43b0-bd95-ead043376d8f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295\" (UID: \"f55f875c-cde6-43b0-bd95-ead043376d8f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"
Oct 30 19:55:59.686107 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:59.686085    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh"
Oct 30 19:55:59.722440 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:59.722414    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6b79\" (UniqueName: \"kubernetes.io/projected/f55f875c-cde6-43b0-bd95-ead043376d8f-kube-api-access-j6b79\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295\" (UID: \"f55f875c-cde6-43b0-bd95-ead043376d8f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"
Oct 30 19:55:59.722563 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:59.722472    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f55f875c-cde6-43b0-bd95-ead043376d8f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295\" (UID: \"f55f875c-cde6-43b0-bd95-ead043376d8f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"
Oct 30 19:55:59.722563 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:59.722501    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f55f875c-cde6-43b0-bd95-ead043376d8f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295\" (UID: \"f55f875c-cde6-43b0-bd95-ead043376d8f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"
Oct 30 19:55:59.722563 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:59.722557    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f55f875c-cde6-43b0-bd95-ead043376d8f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295\" (UID: \"f55f875c-cde6-43b0-bd95-ead043376d8f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"
Oct 30 19:55:59.722710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:59.722601    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f55f875c-cde6-43b0-bd95-ead043376d8f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295\" (UID: \"f55f875c-cde6-43b0-bd95-ead043376d8f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"
Oct 30 19:55:59.722710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:59.722638    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f55f875c-cde6-43b0-bd95-ead043376d8f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295\" (UID: \"f55f875c-cde6-43b0-bd95-ead043376d8f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"
Oct 30 19:55:59.722911 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:59.722890    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f55f875c-cde6-43b0-bd95-ead043376d8f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295\" (UID: \"f55f875c-cde6-43b0-bd95-ead043376d8f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"
Oct 30 19:55:59.722995 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:59.722974    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f55f875c-cde6-43b0-bd95-ead043376d8f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295\" (UID: \"f55f875c-cde6-43b0-bd95-ead043376d8f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"
Oct 30 19:55:59.723046 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:59.722983    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f55f875c-cde6-43b0-bd95-ead043376d8f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295\" (UID: \"f55f875c-cde6-43b0-bd95-ead043376d8f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"
Oct 30 19:55:59.725008 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:59.724982    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f55f875c-cde6-43b0-bd95-ead043376d8f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295\" (UID: \"f55f875c-cde6-43b0-bd95-ead043376d8f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"
Oct 30 19:55:59.725162 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:59.725143    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f55f875c-cde6-43b0-bd95-ead043376d8f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295\" (UID: \"f55f875c-cde6-43b0-bd95-ead043376d8f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"
Oct 30 19:55:59.783696 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:59.783648    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6b79\" (UniqueName: \"kubernetes.io/projected/f55f875c-cde6-43b0-bd95-ead043376d8f-kube-api-access-j6b79\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295\" (UID: \"f55f875c-cde6-43b0-bd95-ead043376d8f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"
Oct 30 19:55:59.807440 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:55:59.807419    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"
Oct 30 19:56:00.182873 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:00.182846    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"]
Oct 30 19:56:00.185987 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:56:00.185962    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55f875c_cde6_43b0_bd95_ead043376d8f.slice/crio-44597a7c42e198f0f1802d9b016bcc84af31f4cd0f1578a8b8f437347edd4971 WatchSource:0}: Error finding container 44597a7c42e198f0f1802d9b016bcc84af31f4cd0f1578a8b8f437347edd4971: Status 404 returned error can't find the container with id 44597a7c42e198f0f1802d9b016bcc84af31f4cd0f1578a8b8f437347edd4971
Oct 30 19:56:00.681304 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:00.681277    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295" event={"ID":"f55f875c-cde6-43b0-bd95-ead043376d8f","Type":"ContainerStarted","Data":"44597a7c42e198f0f1802d9b016bcc84af31f4cd0f1578a8b8f437347edd4971"}
Oct 30 19:56:01.687990 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:01.687952    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295" event={"ID":"f55f875c-cde6-43b0-bd95-ead043376d8f","Type":"ContainerStarted","Data":"e21ddd6b249dc4198c4847f990def7c01415c704ba09b4b3d0ef6ce95154f9f7"}
Oct 30 19:56:01.807837 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:01.807801    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"
Oct 30 19:56:01.822542 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:01.822476    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295" podStartSLOduration=2.105179636 podStartE2EDuration="2.822444026s" totalImagesPullingTime="717.26439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:01.82087809 +0000 UTC m=+800.553093480" watchObservedRunningTime="2026-10-30 19:56:01.822444026 +0000 UTC m=+800.554659409"
Oct 30 19:56:06.604495 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:56:06.604434    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:56:07.397208 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:07.397180    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"]
Oct 30 19:56:07.400743 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:07.400727    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 19:56:07.484167 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:07.484137    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e20c97b-66ce-485b-ba8d-90c2d9e1fa76-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n\" (UID: \"6e20c97b-66ce-485b-ba8d-90c2d9e1fa76\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 19:56:07.484291 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:07.484187    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e20c97b-66ce-485b-ba8d-90c2d9e1fa76-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n\" (UID: \"6e20c97b-66ce-485b-ba8d-90c2d9e1fa76\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 19:56:07.484291 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:07.484206    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e20c97b-66ce-485b-ba8d-90c2d9e1fa76-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n\" (UID: \"6e20c97b-66ce-485b-ba8d-90c2d9e1fa76\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 19:56:07.484362 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:07.484303    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e20c97b-66ce-485b-ba8d-90c2d9e1fa76-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n\" (UID: \"6e20c97b-66ce-485b-ba8d-90c2d9e1fa76\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 19:56:07.484362 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:07.484340    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e20c97b-66ce-485b-ba8d-90c2d9e1fa76-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n\" (UID: \"6e20c97b-66ce-485b-ba8d-90c2d9e1fa76\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 19:56:07.484421 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:07.484393    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq27s\" (UniqueName: \"kubernetes.io/projected/6e20c97b-66ce-485b-ba8d-90c2d9e1fa76-kube-api-access-hq27s\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n\" (UID: \"6e20c97b-66ce-485b-ba8d-90c2d9e1fa76\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 19:56:07.505016 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:07.504993    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 19:56:07.506188 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:07.506165    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"]
Oct 30 19:56:07.585149 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:07.585120    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e20c97b-66ce-485b-ba8d-90c2d9e1fa76-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n\" (UID: \"6e20c97b-66ce-485b-ba8d-90c2d9e1fa76\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 19:56:07.585264 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:07.585153    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e20c97b-66ce-485b-ba8d-90c2d9e1fa76-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n\" (UID: \"6e20c97b-66ce-485b-ba8d-90c2d9e1fa76\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 19:56:07.585264 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:07.585195    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e20c97b-66ce-485b-ba8d-90c2d9e1fa76-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n\" (UID: \"6e20c97b-66ce-485b-ba8d-90c2d9e1fa76\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 19:56:07.585264 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:07.585249    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e20c97b-66ce-485b-ba8d-90c2d9e1fa76-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n\" (UID: \"6e20c97b-66ce-485b-ba8d-90c2d9e1fa76\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 19:56:07.585422 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:07.585295    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq27s\" (UniqueName: \"kubernetes.io/projected/6e20c97b-66ce-485b-ba8d-90c2d9e1fa76-kube-api-access-hq27s\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n\" (UID: \"6e20c97b-66ce-485b-ba8d-90c2d9e1fa76\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 19:56:07.585422 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:07.585367    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e20c97b-66ce-485b-ba8d-90c2d9e1fa76-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n\" (UID: \"6e20c97b-66ce-485b-ba8d-90c2d9e1fa76\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 19:56:07.585639 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:07.585618    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e20c97b-66ce-485b-ba8d-90c2d9e1fa76-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n\" (UID: \"6e20c97b-66ce-485b-ba8d-90c2d9e1fa76\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 19:56:07.585722 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:07.585645    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e20c97b-66ce-485b-ba8d-90c2d9e1fa76-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n\" (UID: \"6e20c97b-66ce-485b-ba8d-90c2d9e1fa76\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 19:56:07.585786 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:07.585739    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e20c97b-66ce-485b-ba8d-90c2d9e1fa76-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n\" (UID: \"6e20c97b-66ce-485b-ba8d-90c2d9e1fa76\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 19:56:07.587415 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:07.587396    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e20c97b-66ce-485b-ba8d-90c2d9e1fa76-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n\" (UID: \"6e20c97b-66ce-485b-ba8d-90c2d9e1fa76\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 19:56:07.587961 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:07.587936    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e20c97b-66ce-485b-ba8d-90c2d9e1fa76-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n\" (UID: \"6e20c97b-66ce-485b-ba8d-90c2d9e1fa76\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 19:56:07.646986 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:07.646963    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq27s\" (UniqueName: \"kubernetes.io/projected/6e20c97b-66ce-485b-ba8d-90c2d9e1fa76-kube-api-access-hq27s\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n\" (UID: \"6e20c97b-66ce-485b-ba8d-90c2d9e1fa76\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 19:56:07.711431 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:07.711365    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 19:56:08.084689 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:08.084656    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"]
Oct 30 19:56:08.088264 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:56:08.088237    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e20c97b_66ce_485b_ba8d_90c2d9e1fa76.slice/crio-1e5203e16b68ba63e2a4f68b0116db6f1fc7630b2111b5ebadc5a96aaff61005 WatchSource:0}: Error finding container 1e5203e16b68ba63e2a4f68b0116db6f1fc7630b2111b5ebadc5a96aaff61005: Status 404 returned error can't find the container with id 1e5203e16b68ba63e2a4f68b0116db6f1fc7630b2111b5ebadc5a96aaff61005
Oct 30 19:56:08.718916 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:08.718888    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" event={"ID":"6e20c97b-66ce-485b-ba8d-90c2d9e1fa76","Type":"ContainerStarted","Data":"1e5203e16b68ba63e2a4f68b0116db6f1fc7630b2111b5ebadc5a96aaff61005"}
Oct 30 19:56:09.724930 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:09.724894    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" event={"ID":"6e20c97b-66ce-485b-ba8d-90c2d9e1fa76","Type":"ContainerStarted","Data":"363d8a46ebf611a15fea5248ce54dc93f48f7c51a544ab86688f6557aaffbd82"}
Oct 30 19:56:09.808322 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:09.808298    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"
Oct 30 19:56:09.820539 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:09.820518    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"
Oct 30 19:56:09.844069 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:09.844025    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" podStartSLOduration=2.147119032 podStartE2EDuration="2.844013932s" totalImagesPullingTime="696.8949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:09.842848196 +0000 UTC m=+808.575063587" watchObservedRunningTime="2026-10-30 19:56:09.844013932 +0000 UTC m=+808.576229321"
Oct 30 19:56:10.712152 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:10.712126    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 19:56:10.740576 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:10.740544    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295"
Oct 30 19:56:17.712090 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:17.712051    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 19:56:17.724215 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:17.724186    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 19:56:17.770764 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:17.770735    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 19:56:31.952614 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:31.952582    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"]
Oct 30 19:56:31.955887 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:31.955871    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.010924 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.010903    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 19:56:32.060577 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.060557    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-66kfh\""
Oct 30 19:56:32.063186 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.063165    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"]
Oct 30 19:56:32.088559 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.088537    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922b8b03-c82e-4d30-9de5-e5d35fe25886-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.088655 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.088567    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.088655 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.088600    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvsbr\" (UniqueName: \"kubernetes.io/projected/922b8b03-c82e-4d30-9de5-e5d35fe25886-kube-api-access-tvsbr\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.088756 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.088738    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.088794 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.088766    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.088794 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.088786    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.088889 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.088878    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.088924 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.088899    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.088924 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.088915    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.190180 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.190152    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.190336 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.190186    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.190336 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.190201    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.190336 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.190223    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.190336 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.190247    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.190336 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.190269    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.190336 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.190330    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922b8b03-c82e-4d30-9de5-e5d35fe25886-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.190763 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.190356    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.190763 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.190409    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvsbr\" (UniqueName: \"kubernetes.io/projected/922b8b03-c82e-4d30-9de5-e5d35fe25886-kube-api-access-tvsbr\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.190763 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.190631    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.190924 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.190800    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.190924 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.190888    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.191034 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.190951    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.191089 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.191057    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922b8b03-c82e-4d30-9de5-e5d35fe25886-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.192655 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.192631    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.193309 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.193281    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.251748 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.251694    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.251956 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.251940    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvsbr\" (UniqueName: \"kubernetes.io/projected/922b8b03-c82e-4d30-9de5-e5d35fe25886-kube-api-access-tvsbr\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-lxggm\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.267977 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.267958    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:32.690232 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.690208    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"]
Oct 30 19:56:32.692337 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:56:32.692308    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922b8b03_c82e_4d30_9de5_e5d35fe25886.slice/crio-846c0e4e213f48a1870c550c0fdb303acc8a01a0d51e047b2289c2f4832187d1 WatchSource:0}: Error finding container 846c0e4e213f48a1870c550c0fdb303acc8a01a0d51e047b2289c2f4832187d1: Status 404 returned error can't find the container with id 846c0e4e213f48a1870c550c0fdb303acc8a01a0d51e047b2289c2f4832187d1
Oct 30 19:56:32.825681 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:32.825653    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm" event={"ID":"922b8b03-c82e-4d30-9de5-e5d35fe25886","Type":"ContainerStarted","Data":"846c0e4e213f48a1870c550c0fdb303acc8a01a0d51e047b2289c2f4832187d1"}
Oct 30 19:56:36.005406 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.005372    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-79bfbfd85f-tvtpm"]
Oct 30 19:56:36.012566 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.012545    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79bfbfd85f-tvtpm"
Oct 30 19:56:36.066694 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.066667    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79bfbfd85f-tvtpm"]
Oct 30 19:56:36.124979 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.124940    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/611350ed-85e8-4bfd-8501-aedf22c38ab6-tls-cert\") pod \"authorino-79bfbfd85f-tvtpm\" (UID: \"611350ed-85e8-4bfd-8501-aedf22c38ab6\") " pod="kuadrant-system/authorino-79bfbfd85f-tvtpm"
Oct 30 19:56:36.125156 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.125005    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gg67\" (UniqueName: \"kubernetes.io/projected/611350ed-85e8-4bfd-8501-aedf22c38ab6-kube-api-access-4gg67\") pod \"authorino-79bfbfd85f-tvtpm\" (UID: \"611350ed-85e8-4bfd-8501-aedf22c38ab6\") " pod="kuadrant-system/authorino-79bfbfd85f-tvtpm"
Oct 30 19:56:36.205441 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.205408    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-8jvqz"]
Oct 30 19:56:36.209244 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.209217    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-8jvqz"
Oct 30 19:56:36.226585 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.226552    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gg67\" (UniqueName: \"kubernetes.io/projected/611350ed-85e8-4bfd-8501-aedf22c38ab6-kube-api-access-4gg67\") pod \"authorino-79bfbfd85f-tvtpm\" (UID: \"611350ed-85e8-4bfd-8501-aedf22c38ab6\") " pod="kuadrant-system/authorino-79bfbfd85f-tvtpm"
Oct 30 19:56:36.226735 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.226686    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/611350ed-85e8-4bfd-8501-aedf22c38ab6-tls-cert\") pod \"authorino-79bfbfd85f-tvtpm\" (UID: \"611350ed-85e8-4bfd-8501-aedf22c38ab6\") " pod="kuadrant-system/authorino-79bfbfd85f-tvtpm"
Oct 30 19:56:36.229381 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.229356    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/611350ed-85e8-4bfd-8501-aedf22c38ab6-tls-cert\") pod \"authorino-79bfbfd85f-tvtpm\" (UID: \"611350ed-85e8-4bfd-8501-aedf22c38ab6\") " pod="kuadrant-system/authorino-79bfbfd85f-tvtpm"
Oct 30 19:56:36.262372 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.262291    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 19:56:36.262540 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.262521    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 19:56:36.262602 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.262524    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-csfbt\""
Oct 30 19:56:36.269159 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.269118    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-8jvqz"]
Oct 30 19:56:36.288613 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.288585    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gg67\" (UniqueName: \"kubernetes.io/projected/611350ed-85e8-4bfd-8501-aedf22c38ab6-kube-api-access-4gg67\") pod \"authorino-79bfbfd85f-tvtpm\" (UID: \"611350ed-85e8-4bfd-8501-aedf22c38ab6\") " pod="kuadrant-system/authorino-79bfbfd85f-tvtpm"
Oct 30 19:56:36.325904 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.325869    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79bfbfd85f-tvtpm"
Oct 30 19:56:36.327242 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.327214    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g254\" (UniqueName: \"kubernetes.io/projected/ae09d202-8b49-4d8c-8a9b-6d432ff5236d-kube-api-access-5g254\") pod \"ai-gateway-controller-5d9cd99f45-8jvqz\" (UID: \"ae09d202-8b49-4d8c-8a9b-6d432ff5236d\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-8jvqz"
Oct 30 19:56:36.327376 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.327252    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae09d202-8b49-4d8c-8a9b-6d432ff5236d-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-8jvqz\" (UID: \"ae09d202-8b49-4d8c-8a9b-6d432ff5236d\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-8jvqz"
Oct 30 19:56:36.428156 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.428099    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g254\" (UniqueName: \"kubernetes.io/projected/ae09d202-8b49-4d8c-8a9b-6d432ff5236d-kube-api-access-5g254\") pod \"ai-gateway-controller-5d9cd99f45-8jvqz\" (UID: \"ae09d202-8b49-4d8c-8a9b-6d432ff5236d\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-8jvqz"
Oct 30 19:56:36.428325 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.428164    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae09d202-8b49-4d8c-8a9b-6d432ff5236d-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-8jvqz\" (UID: \"ae09d202-8b49-4d8c-8a9b-6d432ff5236d\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-8jvqz"
Oct 30 19:56:36.431065 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.431042    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae09d202-8b49-4d8c-8a9b-6d432ff5236d-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-8jvqz\" (UID: \"ae09d202-8b49-4d8c-8a9b-6d432ff5236d\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-8jvqz"
Oct 30 19:56:36.490027 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.489998    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g254\" (UniqueName: \"kubernetes.io/projected/ae09d202-8b49-4d8c-8a9b-6d432ff5236d-kube-api-access-5g254\") pod \"ai-gateway-controller-5d9cd99f45-8jvqz\" (UID: \"ae09d202-8b49-4d8c-8a9b-6d432ff5236d\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-8jvqz"
Oct 30 19:56:36.523821 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.523747    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-8jvqz"
Oct 30 19:56:36.614933 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:56:36.614808    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:56:36.713539 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.713437    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79bfbfd85f-tvtpm"]
Oct 30 19:56:36.721335 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:56:36.721312    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod611350ed_85e8_4bfd_8501_aedf22c38ab6.slice/crio-c10a1a101fe6b21a8b92667ae439e695e26d58ceb1f3258f37ecdcd8df92736c WatchSource:0}: Error finding container c10a1a101fe6b21a8b92667ae439e695e26d58ceb1f3258f37ecdcd8df92736c: Status 404 returned error can't find the container with id c10a1a101fe6b21a8b92667ae439e695e26d58ceb1f3258f37ecdcd8df92736c
Oct 30 19:56:36.732446 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.732417    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:56:36.732527 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.732498    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:56:36.732563 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.732527    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:56:36.848361 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.848331    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79bfbfd85f-tvtpm" event={"ID":"611350ed-85e8-4bfd-8501-aedf22c38ab6","Type":"ContainerStarted","Data":"c10a1a101fe6b21a8b92667ae439e695e26d58ceb1f3258f37ecdcd8df92736c"}
Oct 30 19:56:36.850002 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.849977    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm" event={"ID":"922b8b03-c82e-4d30-9de5-e5d35fe25886","Type":"ContainerStarted","Data":"49bedcad7053981bfeba7feaf7ca48473539c7a890b3a5896b6b9e24125e0a60"}
Oct 30 19:56:36.917944 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.917917    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-8jvqz"]
Oct 30 19:56:36.921433 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:56:36.921407    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae09d202_8b49_4d8c_8a9b_6d432ff5236d.slice/crio-c65d6eae205011ade30401ccd4bf5fca3c80e32cf51c04ff413f75333a4adc1f WatchSource:0}: Error finding container c65d6eae205011ade30401ccd4bf5fca3c80e32cf51c04ff413f75333a4adc1f: Status 404 returned error can't find the container with id c65d6eae205011ade30401ccd4bf5fca3c80e32cf51c04ff413f75333a4adc1f
Oct 30 19:56:36.972229 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:36.972155    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm" podStartSLOduration=1.9347040340000001 podStartE2EDuration="5.972136878s" totalImagesPullingTime="4.037432844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:36.970784516 +0000 UTC m=+835.702999909" watchObservedRunningTime="2026-10-30 19:56:36.972136878 +0000 UTC m=+835.704352271"
Oct 30 19:56:37.268237 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:37.268194    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:37.269730 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:37.269700    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-lxggm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:56:37.269868 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:37.269749    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm" podUID="922b8b03-c82e-4d30-9de5-e5d35fe25886" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 19:56:37.866354 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:37.866315    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-8jvqz" event={"ID":"ae09d202-8b49-4d8c-8a9b-6d432ff5236d","Type":"ContainerStarted","Data":"c65d6eae205011ade30401ccd4bf5fca3c80e32cf51c04ff413f75333a4adc1f"}
Oct 30 19:56:38.269346 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:38.269272    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-lxggm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:56:38.269346 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:38.269327    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm" podUID="922b8b03-c82e-4d30-9de5-e5d35fe25886" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 19:56:38.880095 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:38.880048    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79bfbfd85f-tvtpm" event={"ID":"611350ed-85e8-4bfd-8501-aedf22c38ab6","Type":"ContainerStarted","Data":"d9fee5b5e66ee4d9f5bc3a0478331b53732014fbcfda083fbe74a87574853485"}
Oct 30 19:56:39.015199 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:39.014232    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-79bfbfd85f-tvtpm" podStartSLOduration=2.352178887 podStartE2EDuration="4.014212361s" totalImagesPullingTime="1.662033474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:39.013620851 +0000 UTC m=+837.745836243" watchObservedRunningTime="2026-10-30 19:56:39.014212361 +0000 UTC m=+837.746427752"
Oct 30 19:56:39.069941 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:39.069907    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d4fcd4d-7nxzf"]
Oct 30 19:56:39.070131 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:39.070088    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-75d4fcd4d-7nxzf" podUID="ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc" containerName="authorino" containerID="cri-o://f4b2c5c7dc81d319b40177f2b229fe53b9b29f02bbd56aae69b8b69b2c3678f4" gracePeriod=30
Oct 30 19:56:39.273503 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:39.268612    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-lxggm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:56:39.273503 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:39.268671    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm" podUID="922b8b03-c82e-4d30-9de5-e5d35fe25886" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 19:56:39.377024 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:39.376808    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d4fcd4d-7nxzf"
Oct 30 19:56:39.479651 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:39.479547    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc-tls-cert\" (UniqueName: \"kubernetes.io/secret/ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc-tls-cert\") pod \"ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc\" (UID: \"ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc\") "
Oct 30 19:56:39.479803 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:39.479722    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc-kube-api-access-ggtdr\" (UniqueName: \"kubernetes.io/projected/ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc-kube-api-access-ggtdr\") pod \"ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc\" (UID: \"ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc\") "
Oct 30 19:56:39.483902 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:39.483872    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc-kube-api-access-ggtdr" pod "ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc" (UID: "ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc"). InnerVolumeSpecName "kube-api-access-ggtdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:39.492391 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:39.492364    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc-tls-cert" pod "ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc" (UID: "ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:39.582113 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:39.582042    5958 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc-tls-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:39.582113 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:39.582082    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggtdr\" (UniqueName: \"kubernetes.io/projected/ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc-kube-api-access-ggtdr\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:39.889503 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:39.888787    5958 generic.go:356] "Generic (PLEG): container finished" podID="ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc" containerID="f4b2c5c7dc81d319b40177f2b229fe53b9b29f02bbd56aae69b8b69b2c3678f4" exitCode=0
Oct 30 19:56:39.889503 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:39.889320    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d4fcd4d-7nxzf"
Oct 30 19:56:39.889892 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:39.889749    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d4fcd4d-7nxzf" event={"ID":"ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc","Type":"ContainerDied","Data":"f4b2c5c7dc81d319b40177f2b229fe53b9b29f02bbd56aae69b8b69b2c3678f4"}
Oct 30 19:56:39.889892 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:39.889806    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d4fcd4d-7nxzf" event={"ID":"ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc","Type":"ContainerDied","Data":"5f1134273045d89a7f673f222842d37d51d49f5270aa16fa4d28c4a2d32f5d21"}
Oct 30 19:56:39.889892 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:39.889829    5958 scope.go:122] "RemoveContainer" containerID="f4b2c5c7dc81d319b40177f2b229fe53b9b29f02bbd56aae69b8b69b2c3678f4"
Oct 30 19:56:39.906966 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:39.906939    5958 scope.go:122] "RemoveContainer" containerID="f4b2c5c7dc81d319b40177f2b229fe53b9b29f02bbd56aae69b8b69b2c3678f4"
Oct 30 19:56:39.907303 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:56:39.907282    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b2c5c7dc81d319b40177f2b229fe53b9b29f02bbd56aae69b8b69b2c3678f4\": container with ID starting with f4b2c5c7dc81d319b40177f2b229fe53b9b29f02bbd56aae69b8b69b2c3678f4 not found: ID does not exist" containerID="f4b2c5c7dc81d319b40177f2b229fe53b9b29f02bbd56aae69b8b69b2c3678f4"
Oct 30 19:56:39.907384 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:39.907314    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b2c5c7dc81d319b40177f2b229fe53b9b29f02bbd56aae69b8b69b2c3678f4"} err="failed to get container status \"f4b2c5c7dc81d319b40177f2b229fe53b9b29f02bbd56aae69b8b69b2c3678f4\": rpc error: code = NotFound desc = could not find container \"f4b2c5c7dc81d319b40177f2b229fe53b9b29f02bbd56aae69b8b69b2c3678f4\": container with ID starting with f4b2c5c7dc81d319b40177f2b229fe53b9b29f02bbd56aae69b8b69b2c3678f4 not found: ID does not exist"
Oct 30 19:56:40.070171 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:40.070140    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d4fcd4d-7nxzf"]
Oct 30 19:56:40.120289 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:40.120252    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d4fcd4d-7nxzf"]
Oct 30 19:56:40.268704 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:40.268615    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-lxggm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:56:40.268704 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:40.268674    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm" podUID="922b8b03-c82e-4d30-9de5-e5d35fe25886" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 19:56:41.269121 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:41.268971    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-lxggm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:56:41.269121 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:41.269038    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm" podUID="922b8b03-c82e-4d30-9de5-e5d35fe25886" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 19:56:41.398161 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:41.398129    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc" path="/var/lib/kubelet/pods/ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc/volumes"
Oct 30 19:56:42.270483 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:42.269228    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-lxggm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:56:42.270483 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:42.269287    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm" podUID="922b8b03-c82e-4d30-9de5-e5d35fe25886" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 19:56:43.215024 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:43.214992    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 19:56:43.268494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:43.268305    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-lxggm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:56:43.268494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:43.268355    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm" podUID="922b8b03-c82e-4d30-9de5-e5d35fe25886" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 19:56:43.912203 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:43.912154    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-8jvqz" event={"ID":"ae09d202-8b49-4d8c-8a9b-6d432ff5236d","Type":"ContainerStarted","Data":"a949f49e8462ac3d628e0633a0c9abcf75d0945421ed95682f460fca2c834ba2"}
Oct 30 19:56:43.912713 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:43.912299    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d9cd99f45-8jvqz"
Oct 30 19:56:44.044639 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:44.044576    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d9cd99f45-8jvqz" podStartSLOduration=1.804330312 podStartE2EDuration="8.044558977s" totalImagesPullingTime="6.240228665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:44.039743514 +0000 UTC m=+842.771958902" watchObservedRunningTime="2026-10-30 19:56:44.044558977 +0000 UTC m=+842.776774369"
Oct 30 19:56:44.995528 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:44.995498    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:44.996039 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:44.996003    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:44.996683 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:44.996661    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 19:56:46.557735 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:46.557688    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-27d65"]
Oct 30 19:56:46.558667 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:46.558643    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc" containerName="authorino"
Oct 30 19:56:46.558667 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:46.558666    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc" containerName="authorino"
Oct 30 19:56:46.558872 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:46.558859    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae195e92-06b3-49b5-a4d9-e6a9bd08ebdc" containerName="authorino"
Oct 30 19:56:46.563299 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:46.563275    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-27d65"
Oct 30 19:56:46.617700 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:46.617673    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-27d65"]
Oct 30 19:56:46.653348 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:46.653321    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxn29\" (UniqueName: \"kubernetes.io/projected/7bb44edd-1fa8-495d-872f-84e0156aedf0-kube-api-access-qxn29\") pod \"payload-pre-processing-7b98bb678c-27d65\" (UID: \"7bb44edd-1fa8-495d-872f-84e0156aedf0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-27d65"
Oct 30 19:56:46.653533 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:46.653376    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb44edd-1fa8-495d-872f-84e0156aedf0-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-27d65\" (UID: \"7bb44edd-1fa8-495d-872f-84e0156aedf0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-27d65"
Oct 30 19:56:46.665792 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:46.665766    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 19:56:46.665913 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:46.665803    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-l5hs8\""
Oct 30 19:56:46.717151 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:46.717114    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-mlhzl"]
Oct 30 19:56:46.720801 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:46.720778    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-mlhzl"
Oct 30 19:56:46.754793 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:46.754755    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxn29\" (UniqueName: \"kubernetes.io/projected/7bb44edd-1fa8-495d-872f-84e0156aedf0-kube-api-access-qxn29\") pod \"payload-pre-processing-7b98bb678c-27d65\" (UID: \"7bb44edd-1fa8-495d-872f-84e0156aedf0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-27d65"
Oct 30 19:56:46.754937 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:46.754864    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb44edd-1fa8-495d-872f-84e0156aedf0-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-27d65\" (UID: \"7bb44edd-1fa8-495d-872f-84e0156aedf0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-27d65"
Oct 30 19:56:46.755593 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:46.755574    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bb44edd-1fa8-495d-872f-84e0156aedf0-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-27d65\" (UID: \"7bb44edd-1fa8-495d-872f-84e0156aedf0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-27d65"
Oct 30 19:56:46.760238 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:46.760200    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-mlhzl"]
Oct 30 19:56:46.816134 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:46.816108    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxn29\" (UniqueName: \"kubernetes.io/projected/7bb44edd-1fa8-495d-872f-84e0156aedf0-kube-api-access-qxn29\") pod \"payload-pre-processing-7b98bb678c-27d65\" (UID: \"7bb44edd-1fa8-495d-872f-84e0156aedf0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-27d65"
Oct 30 19:56:46.856065 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:46.856034    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcjff\" (UniqueName: \"kubernetes.io/projected/86400592-9707-4383-bac2-3f7d8e8eb053-kube-api-access-gcjff\") pod \"payload-processing-5f78687d5b-mlhzl\" (UID: \"86400592-9707-4383-bac2-3f7d8e8eb053\") " pod="openshift-ingress/payload-processing-5f78687d5b-mlhzl"
Oct 30 19:56:46.856177 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:46.856068    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86400592-9707-4383-bac2-3f7d8e8eb053-plugins-config-volume\") pod \"payload-processing-5f78687d5b-mlhzl\" (UID: \"86400592-9707-4383-bac2-3f7d8e8eb053\") " pod="openshift-ingress/payload-processing-5f78687d5b-mlhzl"
Oct 30 19:56:46.875777 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:46.875755    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-27d65"
Oct 30 19:56:46.957773 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:46.957742    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcjff\" (UniqueName: \"kubernetes.io/projected/86400592-9707-4383-bac2-3f7d8e8eb053-kube-api-access-gcjff\") pod \"payload-processing-5f78687d5b-mlhzl\" (UID: \"86400592-9707-4383-bac2-3f7d8e8eb053\") " pod="openshift-ingress/payload-processing-5f78687d5b-mlhzl"
Oct 30 19:56:46.957899 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:46.957783    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86400592-9707-4383-bac2-3f7d8e8eb053-plugins-config-volume\") pod \"payload-processing-5f78687d5b-mlhzl\" (UID: \"86400592-9707-4383-bac2-3f7d8e8eb053\") " pod="openshift-ingress/payload-processing-5f78687d5b-mlhzl"
Oct 30 19:56:46.958541 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:46.958521    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86400592-9707-4383-bac2-3f7d8e8eb053-plugins-config-volume\") pod \"payload-processing-5f78687d5b-mlhzl\" (UID: \"86400592-9707-4383-bac2-3f7d8e8eb053\") " pod="openshift-ingress/payload-processing-5f78687d5b-mlhzl"
Oct 30 19:56:47.019294 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:47.019263    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcjff\" (UniqueName: \"kubernetes.io/projected/86400592-9707-4383-bac2-3f7d8e8eb053-kube-api-access-gcjff\") pod \"payload-processing-5f78687d5b-mlhzl\" (UID: \"86400592-9707-4383-bac2-3f7d8e8eb053\") " pod="openshift-ingress/payload-processing-5f78687d5b-mlhzl"
Oct 30 19:56:47.034143 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:47.034120    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-mlhzl"
Oct 30 19:56:47.249393 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:47.249361    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-27d65"]
Oct 30 19:56:47.253020 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:56:47.252991    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb44edd_1fa8_495d_872f_84e0156aedf0.slice/crio-6c5ea9b1bff800f8e4e9cf4fa7f3306f35d7d78d0cf1f45a900e3a122c0282db WatchSource:0}: Error finding container 6c5ea9b1bff800f8e4e9cf4fa7f3306f35d7d78d0cf1f45a900e3a122c0282db: Status 404 returned error can't find the container with id 6c5ea9b1bff800f8e4e9cf4fa7f3306f35d7d78d0cf1f45a900e3a122c0282db
Oct 30 19:56:47.406426 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:47.406336    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-mlhzl"]
Oct 30 19:56:47.409838 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:56:47.409808    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86400592_9707_4383_bac2_3f7d8e8eb053.slice/crio-59186bd5bf27d9d7b701d2d4fc058cc51b0938409b897174869428e32d2d336a WatchSource:0}: Error finding container 59186bd5bf27d9d7b701d2d4fc058cc51b0938409b897174869428e32d2d336a: Status 404 returned error can't find the container with id 59186bd5bf27d9d7b701d2d4fc058cc51b0938409b897174869428e32d2d336a
Oct 30 19:56:47.935074 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:47.935043    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-mlhzl" event={"ID":"86400592-9707-4383-bac2-3f7d8e8eb053","Type":"ContainerStarted","Data":"59186bd5bf27d9d7b701d2d4fc058cc51b0938409b897174869428e32d2d336a"}
Oct 30 19:56:47.936373 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:47.936342    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-27d65" event={"ID":"7bb44edd-1fa8-495d-872f-84e0156aedf0","Type":"ContainerStarted","Data":"6c5ea9b1bff800f8e4e9cf4fa7f3306f35d7d78d0cf1f45a900e3a122c0282db"}
Oct 30 19:56:50.953178 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:50.953141    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-27d65" event={"ID":"7bb44edd-1fa8-495d-872f-84e0156aedf0","Type":"ContainerStarted","Data":"c369102cf38ac66b6d0e0c10dc4b4ab4bc7ea49385ae5f833f9a33acccb905da"}
Oct 30 19:56:50.953585 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:50.953207    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-27d65"
Oct 30 19:56:50.954513 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:50.954491    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-mlhzl" event={"ID":"86400592-9707-4383-bac2-3f7d8e8eb053","Type":"ContainerStarted","Data":"8874cbe3ff1ddffa9f7140c3dbca7db7c559b7d48a0c54bfb60e1af741e4100d"}
Oct 30 19:56:50.954618 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:50.954604    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-mlhzl"
Oct 30 19:56:51.069759 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:51.069687    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-27d65" podStartSLOduration=2.168728006 podStartE2EDuration="5.06916905s" totalImagesPullingTime="2.900441044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:51.069094028 +0000 UTC m=+849.801309419" watchObservedRunningTime="2026-10-30 19:56:51.06916905 +0000 UTC m=+849.801384432"
Oct 30 19:56:51.187036 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:51.186971    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-mlhzl" podStartSLOduration=2.4026832750000002 podStartE2EDuration="5.186953078s" totalImagesPullingTime="2.784269803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:51.185970253 +0000 UTC m=+849.918185643" watchObservedRunningTime="2026-10-30 19:56:51.186953078 +0000 UTC m=+849.919168467"
Oct 30 19:56:54.922713 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:54.922673    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d9cd99f45-8jvqz"
Oct 30 19:56:55.294694 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:55.294616    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-dd496665f-x67wb"]
Oct 30 19:56:55.298219 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:55.298200    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-dd496665f-x67wb"
Oct 30 19:56:55.355891 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:55.355866    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-dd496665f-x67wb"]
Oct 30 19:56:55.454604 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:55.454580    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07bb7383-2ee5-459c-9464-06668df1a41f-metrics-certs\") pod \"ai-gateway-controller-dd496665f-x67wb\" (UID: \"07bb7383-2ee5-459c-9464-06668df1a41f\") " pod="opendatahub/ai-gateway-controller-dd496665f-x67wb"
Oct 30 19:56:55.454720 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:55.454643    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6lft\" (UniqueName: \"kubernetes.io/projected/07bb7383-2ee5-459c-9464-06668df1a41f-kube-api-access-r6lft\") pod \"ai-gateway-controller-dd496665f-x67wb\" (UID: \"07bb7383-2ee5-459c-9464-06668df1a41f\") " pod="opendatahub/ai-gateway-controller-dd496665f-x67wb"
Oct 30 19:56:55.555292 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:55.555241    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6lft\" (UniqueName: \"kubernetes.io/projected/07bb7383-2ee5-459c-9464-06668df1a41f-kube-api-access-r6lft\") pod \"ai-gateway-controller-dd496665f-x67wb\" (UID: \"07bb7383-2ee5-459c-9464-06668df1a41f\") " pod="opendatahub/ai-gateway-controller-dd496665f-x67wb"
Oct 30 19:56:55.555384 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:55.555301    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07bb7383-2ee5-459c-9464-06668df1a41f-metrics-certs\") pod \"ai-gateway-controller-dd496665f-x67wb\" (UID: \"07bb7383-2ee5-459c-9464-06668df1a41f\") " pod="opendatahub/ai-gateway-controller-dd496665f-x67wb"
Oct 30 19:56:55.557665 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:55.557649    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07bb7383-2ee5-459c-9464-06668df1a41f-metrics-certs\") pod \"ai-gateway-controller-dd496665f-x67wb\" (UID: \"07bb7383-2ee5-459c-9464-06668df1a41f\") " pod="opendatahub/ai-gateway-controller-dd496665f-x67wb"
Oct 30 19:56:55.616590 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:55.616564    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6lft\" (UniqueName: \"kubernetes.io/projected/07bb7383-2ee5-459c-9464-06668df1a41f-kube-api-access-r6lft\") pod \"ai-gateway-controller-dd496665f-x67wb\" (UID: \"07bb7383-2ee5-459c-9464-06668df1a41f\") " pod="opendatahub/ai-gateway-controller-dd496665f-x67wb"
Oct 30 19:56:55.909987 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:55.909961    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-dd496665f-x67wb"
Oct 30 19:56:56.281053 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:56.280979    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-dd496665f-x67wb"]
Oct 30 19:56:56.284613 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:56:56.284578    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07bb7383_2ee5_459c_9464_06668df1a41f.slice/crio-b7582d9f7fbdc8d67b45723752cdbceed502166d312535346585a548ee8fa598 WatchSource:0}: Error finding container b7582d9f7fbdc8d67b45723752cdbceed502166d312535346585a548ee8fa598: Status 404 returned error can't find the container with id b7582d9f7fbdc8d67b45723752cdbceed502166d312535346585a548ee8fa598
Oct 30 19:56:56.986631 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:56.986598    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-dd496665f-x67wb" event={"ID":"07bb7383-2ee5-459c-9464-06668df1a41f","Type":"ContainerStarted","Data":"b7582d9f7fbdc8d67b45723752cdbceed502166d312535346585a548ee8fa598"}
Oct 30 19:56:57.993086 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:57.993049    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-dd496665f-x67wb" event={"ID":"07bb7383-2ee5-459c-9464-06668df1a41f","Type":"ContainerStarted","Data":"585b2ffd96bf06d3ed2ccce66dc1beb1437d049b8b6170f2da2805cb85f48b73"}
Oct 30 19:56:57.993481 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:57.993118    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-dd496665f-x67wb"
Oct 30 19:56:58.110671 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:56:58.110626    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-dd496665f-x67wb" podStartSLOduration=2.277295892 podStartE2EDuration="3.110613329s" totalImagesPullingTime="833.317437ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:58.109434405 +0000 UTC m=+856.841649797" watchObservedRunningTime="2026-10-30 19:56:58.110613329 +0000 UTC m=+856.842828717"
Oct 30 19:57:01.963227 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:01.963203    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-27d65"
Oct 30 19:57:01.963603 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:01.963570    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-mlhzl"
Oct 30 19:57:06.604777 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:57:06.604740    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:57:09.000558 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:09.000528    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-dd496665f-x67wb"
Oct 30 19:57:09.169955 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:09.169921    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-8jvqz"]
Oct 30 19:57:09.170196 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:09.170160    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d9cd99f45-8jvqz" podUID="ae09d202-8b49-4d8c-8a9b-6d432ff5236d" containerName="manager" containerID="cri-o://a949f49e8462ac3d628e0633a0c9abcf75d0945421ed95682f460fca2c834ba2" gracePeriod=30
Oct 30 19:57:09.414683 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:09.414659    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-8jvqz"
Oct 30 19:57:09.485204 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:09.485178    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae09d202-8b49-4d8c-8a9b-6d432ff5236d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae09d202-8b49-4d8c-8a9b-6d432ff5236d-metrics-certs\") pod \"ae09d202-8b49-4d8c-8a9b-6d432ff5236d\" (UID: \"ae09d202-8b49-4d8c-8a9b-6d432ff5236d\") "
Oct 30 19:57:09.485302 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:09.485255    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae09d202-8b49-4d8c-8a9b-6d432ff5236d-kube-api-access-5g254\" (UniqueName: \"kubernetes.io/projected/ae09d202-8b49-4d8c-8a9b-6d432ff5236d-kube-api-access-5g254\") pod \"ae09d202-8b49-4d8c-8a9b-6d432ff5236d\" (UID: \"ae09d202-8b49-4d8c-8a9b-6d432ff5236d\") "
Oct 30 19:57:09.487292 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:09.487266    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae09d202-8b49-4d8c-8a9b-6d432ff5236d-metrics-certs" pod "ae09d202-8b49-4d8c-8a9b-6d432ff5236d" (UID: "ae09d202-8b49-4d8c-8a9b-6d432ff5236d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:57:09.487389 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:09.487350    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae09d202-8b49-4d8c-8a9b-6d432ff5236d-kube-api-access-5g254" pod "ae09d202-8b49-4d8c-8a9b-6d432ff5236d" (UID: "ae09d202-8b49-4d8c-8a9b-6d432ff5236d"). InnerVolumeSpecName "kube-api-access-5g254". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:09.586552 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:09.586531    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g254\" (UniqueName: \"kubernetes.io/projected/ae09d202-8b49-4d8c-8a9b-6d432ff5236d-kube-api-access-5g254\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:57:09.586552 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:09.586552    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae09d202-8b49-4d8c-8a9b-6d432ff5236d-metrics-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:57:10.051662 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:10.051572    5958 generic.go:356] "Generic (PLEG): container finished" podID="ae09d202-8b49-4d8c-8a9b-6d432ff5236d" containerID="a949f49e8462ac3d628e0633a0c9abcf75d0945421ed95682f460fca2c834ba2" exitCode=0
Oct 30 19:57:10.051662 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:10.051634    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-8jvqz"
Oct 30 19:57:10.052084 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:10.051662    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-8jvqz" event={"ID":"ae09d202-8b49-4d8c-8a9b-6d432ff5236d","Type":"ContainerDied","Data":"a949f49e8462ac3d628e0633a0c9abcf75d0945421ed95682f460fca2c834ba2"}
Oct 30 19:57:10.052084 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:10.051739    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-8jvqz" event={"ID":"ae09d202-8b49-4d8c-8a9b-6d432ff5236d","Type":"ContainerDied","Data":"c65d6eae205011ade30401ccd4bf5fca3c80e32cf51c04ff413f75333a4adc1f"}
Oct 30 19:57:10.052084 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:10.051768    5958 scope.go:122] "RemoveContainer" containerID="a949f49e8462ac3d628e0633a0c9abcf75d0945421ed95682f460fca2c834ba2"
Oct 30 19:57:10.061376 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:10.061358    5958 scope.go:122] "RemoveContainer" containerID="a949f49e8462ac3d628e0633a0c9abcf75d0945421ed95682f460fca2c834ba2"
Oct 30 19:57:10.061651 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:57:10.061630    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a949f49e8462ac3d628e0633a0c9abcf75d0945421ed95682f460fca2c834ba2\": container with ID starting with a949f49e8462ac3d628e0633a0c9abcf75d0945421ed95682f460fca2c834ba2 not found: ID does not exist" containerID="a949f49e8462ac3d628e0633a0c9abcf75d0945421ed95682f460fca2c834ba2"
Oct 30 19:57:10.061716 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:10.061664    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a949f49e8462ac3d628e0633a0c9abcf75d0945421ed95682f460fca2c834ba2"} err="failed to get container status \"a949f49e8462ac3d628e0633a0c9abcf75d0945421ed95682f460fca2c834ba2\": rpc error: code = NotFound desc = could not find container \"a949f49e8462ac3d628e0633a0c9abcf75d0945421ed95682f460fca2c834ba2\": container with ID starting with a949f49e8462ac3d628e0633a0c9abcf75d0945421ed95682f460fca2c834ba2 not found: ID does not exist"
Oct 30 19:57:10.215596 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:10.215573    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-8jvqz"]
Oct 30 19:57:10.265372 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:10.265353    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-8jvqz"]
Oct 30 19:57:11.397317 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:11.397283    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae09d202-8b49-4d8c-8a9b-6d432ff5236d" path="/var/lib/kubelet/pods/ae09d202-8b49-4d8c-8a9b-6d432ff5236d/volumes"
Oct 30 19:57:36.604952 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:57:36.604915    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:57:41.475621 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.475591    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"]
Oct 30 19:57:41.477133 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.477108    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ae09d202-8b49-4d8c-8a9b-6d432ff5236d" containerName="manager"
Oct 30 19:57:41.477133 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.477133    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae09d202-8b49-4d8c-8a9b-6d432ff5236d" containerName="manager"
Oct 30 19:57:41.477344 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.477214    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae09d202-8b49-4d8c-8a9b-6d432ff5236d" containerName="manager"
Oct 30 19:57:41.482616 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.482592    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.501291 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.501269    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 19:57:41.501477 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.501462    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 19:57:41.536547 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.536524    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-407632-openshift-default-dockercfg-5j7qg\""
Oct 30 19:57:41.575255 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.575216    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jftrw\" (UniqueName: \"kubernetes.io/projected/fde4d62b-bd67-4915-a145-e5511e651c25-kube-api-access-jftrw\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.575366 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.575272    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.575412 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.575368    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.575412 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.575388    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fde4d62b-bd67-4915-a145-e5511e651c25-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.575526 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.575419    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fde4d62b-bd67-4915-a145-e5511e651c25-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.575526 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.575443    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.575627 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.575611    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.575661 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.575649    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fde4d62b-bd67-4915-a145-e5511e651c25-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.575761 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.575740    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.599162 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.599137    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"]
Oct 30 19:57:41.676848 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.676822    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fde4d62b-bd67-4915-a145-e5511e651c25-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.676976 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.676858    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.676976 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.676882    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.676976 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.676904    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fde4d62b-bd67-4915-a145-e5511e651c25-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.676976 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.676935    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.677208 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.676982    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jftrw\" (UniqueName: \"kubernetes.io/projected/fde4d62b-bd67-4915-a145-e5511e651c25-kube-api-access-jftrw\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.677208 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.677017    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.677208 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.677067    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.677208 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.677167    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fde4d62b-bd67-4915-a145-e5511e651c25-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.677556 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.677529    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.677718 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.677537    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.677718 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.677629    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.677718 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.677655    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.677927 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.677910    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fde4d62b-bd67-4915-a145-e5511e651c25-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.679380 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.679359    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fde4d62b-bd67-4915-a145-e5511e651c25-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.679509 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.679472    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.738307 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.738248    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fde4d62b-bd67-4915-a145-e5511e651c25-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.738731 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.738713    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jftrw\" (UniqueName: \"kubernetes.io/projected/fde4d62b-bd67-4915-a145-e5511e651c25-kube-api-access-jftrw\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.798505 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.798483    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:41.809887 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.809866    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"]
Oct 30 19:57:41.813898 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.813881    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:41.874489 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.871835    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-f75272-openshift-default-dockercfg-9blkz\""
Oct 30 19:57:41.880961 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.879649    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:41.880961 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.879719    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:41.880961 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.879749    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm9nh\" (UniqueName: \"kubernetes.io/projected/590c5cad-401a-49be-a4b1-79892dc39d5d-kube-api-access-cm9nh\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:41.880961 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.879804    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:41.880961 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.879835    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:41.880961 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.879896    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:41.880961 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.879924    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:41.880961 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.879963    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:41.880961 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.879991    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/590c5cad-401a-49be-a4b1-79892dc39d5d-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:41.881578 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.881431    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"]
Oct 30 19:57:41.981417 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.981394    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:41.981576 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.981423    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:41.981576 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.981473    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:41.981576 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.981497    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/590c5cad-401a-49be-a4b1-79892dc39d5d-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:41.981711 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.981580    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:41.981711 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.981620    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:41.981711 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.981646    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm9nh\" (UniqueName: \"kubernetes.io/projected/590c5cad-401a-49be-a4b1-79892dc39d5d-kube-api-access-cm9nh\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:41.981711 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.981696    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:41.981903 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.981731    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:41.981903 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.981847    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:41.982075 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.982053    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:41.982121 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.982084    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:41.982200 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.982144    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:41.982281 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.982259    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/590c5cad-401a-49be-a4b1-79892dc39d5d-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:41.983987 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.983966    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:41.984218 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:41.984200    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:42.049491 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.049420    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:42.049644 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.049627    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm9nh\" (UniqueName: \"kubernetes.io/projected/590c5cad-401a-49be-a4b1-79892dc39d5d-kube-api-access-cm9nh\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:42.130036 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.130010    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:42.237834 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.237810    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"]
Oct 30 19:57:42.239832 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:57:42.239806    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde4d62b_bd67_4915_a145_e5511e651c25.slice/crio-08a99d775bb3cf2a4815c760d35c5bddbea0b085db900367d9b5b3884fdd2455 WatchSource:0}: Error finding container 08a99d775bb3cf2a4815c760d35c5bddbea0b085db900367d9b5b3884fdd2455: Status 404 returned error can't find the container with id 08a99d775bb3cf2a4815c760d35c5bddbea0b085db900367d9b5b3884fdd2455
Oct 30 19:57:42.242048 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.242016    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:57:42.242150 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.242084    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:57:42.242150 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.242114    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:57:42.523245 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.523210    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"]
Oct 30 19:57:42.528120 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.528096    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.528389 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.528367    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"]
Oct 30 19:57:42.532060 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:57:42.532039    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod590c5cad_401a_49be_a4b1_79892dc39d5d.slice/crio-4b0fdbc5ed87f86154965f9dd1b7f0e15d5255352127833acf26851224910501 WatchSource:0}: Error finding container 4b0fdbc5ed87f86154965f9dd1b7f0e15d5255352127833acf26851224910501: Status 404 returned error can't find the container with id 4b0fdbc5ed87f86154965f9dd1b7f0e15d5255352127833acf26851224910501
Oct 30 19:57:42.538927 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.536726    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:57:42.538927 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.536822    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:57:42.538927 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.536878    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:57:42.581566 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.581542    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f84d85-openshift-default-dockercfg-lv5m9\""
Oct 30 19:57:42.589647 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.589613    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"]
Oct 30 19:57:42.590241 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.590200    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.590330 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.590242    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.590330 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.590287    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.590411 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.590327    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.590411 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.590357    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpq5x\" (UniqueName: \"kubernetes.io/projected/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-kube-api-access-jpq5x\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.590411 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.590401    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.590596 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.590439    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.590596 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.590549    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.590596 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.590580    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.691759 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.691724    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.691951 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.691774    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.691951 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.691820    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.691951 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.691845    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.691951 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.691884    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.691951 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.691920    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.691951 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.691951    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpq5x\" (UniqueName: \"kubernetes.io/projected/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-kube-api-access-jpq5x\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.692266 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.691990    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.692266 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.692027    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.692266 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.692155    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.692743 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.692720    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.692857 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.692744    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.693059 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.693034    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.693140 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.693096    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.694910 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.694886    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.697025 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.697002    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.758794 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.758766    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.758943 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.758923    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpq5x\" (UniqueName: \"kubernetes.io/projected/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-kube-api-access-jpq5x\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:42.869113 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:42.869080    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:43.178579 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.178515    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"]
Oct 30 19:57:43.183167 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.183142    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.204541 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.204510    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v" event={"ID":"fde4d62b-bd67-4915-a145-e5511e651c25","Type":"ContainerStarted","Data":"5804a85bb63e293e5e14f36db543f910ccec895a02f22cc2da98da5a2fcc5d16"}
Oct 30 19:57:43.204700 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.204565    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v" event={"ID":"fde4d62b-bd67-4915-a145-e5511e651c25","Type":"ContainerStarted","Data":"08a99d775bb3cf2a4815c760d35c5bddbea0b085db900367d9b5b3884fdd2455"}
Oct 30 19:57:43.206954 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.206917    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg" event={"ID":"590c5cad-401a-49be-a4b1-79892dc39d5d","Type":"ContainerStarted","Data":"44b5342dca4dbe6d31592852da154980e2c82c0088a84e20ba8a998573a55bcc"}
Oct 30 19:57:43.207054 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.206964    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg" event={"ID":"590c5cad-401a-49be-a4b1-79892dc39d5d","Type":"ContainerStarted","Data":"4b0fdbc5ed87f86154965f9dd1b7f0e15d5255352127833acf26851224910501"}
Oct 30 19:57:43.240279 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.240256    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-d67ea9-openshift-default-dockercfg-t5857\""
Oct 30 19:57:43.257404 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.257372    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"]
Oct 30 19:57:43.298379 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.298357    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.298508 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.298386    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.298508 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.298417    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56ec387e-8e34-472e-b8f5-032a57c517e9-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.298593 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.298554    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.298636 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.298615    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.298671 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.298653    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.298708 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.298687    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.298944 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.298919    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.299012 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.298969    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxz5c\" (UniqueName: \"kubernetes.io/projected/56ec387e-8e34-472e-b8f5-032a57c517e9-kube-api-access-wxz5c\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.310176 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.310158    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"]
Oct 30 19:57:43.312841 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:57:43.312821    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc60e1a_391d_43b0_8b8e_2f6f33c234be.slice/crio-60ebb0273875ad49adbfa2cdec70b2063eedc4336a9df0d23562e7a5bff8b904 WatchSource:0}: Error finding container 60ebb0273875ad49adbfa2cdec70b2063eedc4336a9df0d23562e7a5bff8b904: Status 404 returned error can't find the container with id 60ebb0273875ad49adbfa2cdec70b2063eedc4336a9df0d23562e7a5bff8b904
Oct 30 19:57:43.314860 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.314830    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:57:43.314934 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.314888    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:57:43.314934 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.314917    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:57:43.399727 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.399702    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.399849 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.399734    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.399849 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.399755    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56ec387e-8e34-472e-b8f5-032a57c517e9-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.399849 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.399780    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.399849 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.399833    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.400058 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.399860    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.400146 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.400125    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.400146 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.400137    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.400266 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.400138    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.400266 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.400194    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.400266 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.400208    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.400266 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.400254    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxz5c\" (UniqueName: \"kubernetes.io/projected/56ec387e-8e34-472e-b8f5-032a57c517e9-kube-api-access-wxz5c\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.400503 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.400389    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.401095 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.401074    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56ec387e-8e34-472e-b8f5-032a57c517e9-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.402626 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.402608    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.403096 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.403076    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.459011 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.458903    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v" podStartSLOduration=2.458889737 podStartE2EDuration="2.458889737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:43.457564468 +0000 UTC m=+902.189779861" watchObservedRunningTime="2026-10-30 19:57:43.458889737 +0000 UTC m=+902.191105126"
Oct 30 19:57:43.464649 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.464622    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxz5c\" (UniqueName: \"kubernetes.io/projected/56ec387e-8e34-472e-b8f5-032a57c517e9-kube-api-access-wxz5c\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.465614 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.465568    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.495917 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.495895    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:43.588729 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.588678    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg" podStartSLOduration=2.588663987 podStartE2EDuration="2.588663987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:43.581019845 +0000 UTC m=+902.313235234" watchObservedRunningTime="2026-10-30 19:57:43.588663987 +0000 UTC m=+902.320879376"
Oct 30 19:57:43.798697 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.798625    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:43.803724 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.803698    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:43.883984 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.883961    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"]
Oct 30 19:57:43.886680 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:57:43.886652    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ec387e_8e34_472e_b8f5_032a57c517e9.slice/crio-9abb42f96a7d49aa41cb7bb2533a8cd17d390f7acfc73341d0316e0fa4cc8556 WatchSource:0}: Error finding container 9abb42f96a7d49aa41cb7bb2533a8cd17d390f7acfc73341d0316e0fa4cc8556: Status 404 returned error can't find the container with id 9abb42f96a7d49aa41cb7bb2533a8cd17d390f7acfc73341d0316e0fa4cc8556
Oct 30 19:57:43.889272 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.889243    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:57:43.889328 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.889304    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:57:43.889364 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:43.889333    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:57:44.130416 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:44.130384    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:44.135515 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:44.135491    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:44.214039 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:44.214004    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw" event={"ID":"56ec387e-8e34-472e-b8f5-032a57c517e9","Type":"ContainerStarted","Data":"a706cb79798838937a181d0a6787ccfddc79c46487fd13d6073c0ea1910bba0c"}
Oct 30 19:57:44.214223 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:44.214067    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw" event={"ID":"56ec387e-8e34-472e-b8f5-032a57c517e9","Type":"ContainerStarted","Data":"9abb42f96a7d49aa41cb7bb2533a8cd17d390f7acfc73341d0316e0fa4cc8556"}
Oct 30 19:57:44.215996 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:44.215966    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6" event={"ID":"6bc60e1a-391d-43b0-8b8e-2f6f33c234be","Type":"ContainerStarted","Data":"e05dc617cb1c861aedefcb6d4afe1c26b2a0a634e86006c15482f836f64e38a8"}
Oct 30 19:57:44.216126 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:44.216017    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6" event={"ID":"6bc60e1a-391d-43b0-8b8e-2f6f33c234be","Type":"ContainerStarted","Data":"60ebb0273875ad49adbfa2cdec70b2063eedc4336a9df0d23562e7a5bff8b904"}
Oct 30 19:57:44.216815 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:44.216372    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:44.216815 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:44.216494    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:44.217158 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:44.217137    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:57:44.217293 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:44.217277    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:57:44.496580 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:44.496501    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:44.501580 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:44.501553    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:44.506802 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:44.506750    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6" podStartSLOduration=2.506732082 podStartE2EDuration="2.506732082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:44.504565759 +0000 UTC m=+903.236781169" watchObservedRunningTime="2026-10-30 19:57:44.506732082 +0000 UTC m=+903.238947474"
Oct 30 19:57:44.758265 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:44.758162    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw" podStartSLOduration=1.7581464310000001 podStartE2EDuration="1.758146431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:44.756225144 +0000 UTC m=+903.488440534" watchObservedRunningTime="2026-10-30 19:57:44.758146431 +0000 UTC m=+903.490361820"
Oct 30 19:57:44.870652 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:44.869842    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:44.880800 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:44.880774    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:45.221892 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:45.221855    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:45.222214 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:45.222187    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:45.222318 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:45.222257    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:57:45.222318 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:45.222295    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:57:52.577709 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:52.577622    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp"]
Oct 30 19:57:52.582597 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:52.582574    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp"
Oct 30 19:57:52.638028 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:52.638007    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-f75272\""
Oct 30 19:57:52.687273 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:52.687236    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-f75272-dockercfg-vh4k4\""
Oct 30 19:57:52.689345 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:52.689322    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp"]
Oct 30 19:57:52.692653 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:52.692612    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w97d\" (UniqueName: \"kubernetes.io/projected/fbd4cbac-918f-4517-bdb7-9c4fcecf2401-kube-api-access-5w97d\") pod \"payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp\" (UID: \"fbd4cbac-918f-4517-bdb7-9c4fcecf2401\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp"
Oct 30 19:57:52.692913 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:52.692892    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd4cbac-918f-4517-bdb7-9c4fcecf2401-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp\" (UID: \"fbd4cbac-918f-4517-bdb7-9c4fcecf2401\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp"
Oct 30 19:57:52.738530 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:52.738498    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb"]
Oct 30 19:57:52.742613 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:52.742594    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb"
Oct 30 19:57:52.782341 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:52.782314    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb"]
Oct 30 19:57:52.794292 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:52.794269    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd4cbac-918f-4517-bdb7-9c4fcecf2401-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp\" (UID: \"fbd4cbac-918f-4517-bdb7-9c4fcecf2401\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp"
Oct 30 19:57:52.794392 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:52.794375    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w97d\" (UniqueName: \"kubernetes.io/projected/fbd4cbac-918f-4517-bdb7-9c4fcecf2401-kube-api-access-5w97d\") pod \"payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp\" (UID: \"fbd4cbac-918f-4517-bdb7-9c4fcecf2401\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp"
Oct 30 19:57:52.795034 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:52.795012    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd4cbac-918f-4517-bdb7-9c4fcecf2401-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp\" (UID: \"fbd4cbac-918f-4517-bdb7-9c4fcecf2401\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp"
Oct 30 19:57:52.859552 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:52.859491    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w97d\" (UniqueName: \"kubernetes.io/projected/fbd4cbac-918f-4517-bdb7-9c4fcecf2401-kube-api-access-5w97d\") pod \"payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp\" (UID: \"fbd4cbac-918f-4517-bdb7-9c4fcecf2401\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp"
Oct 30 19:57:52.895434 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:52.895406    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp"
Oct 30 19:57:52.895665 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:52.895644    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6l5q\" (UniqueName: \"kubernetes.io/projected/6bdc83b4-72fc-42b0-83f9-740a2ce1cb71-kube-api-access-m6l5q\") pod \"payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb\" (UID: \"6bdc83b4-72fc-42b0-83f9-740a2ce1cb71\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb"
Oct 30 19:57:52.895764 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:52.895752    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bdc83b4-72fc-42b0-83f9-740a2ce1cb71-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb\" (UID: \"6bdc83b4-72fc-42b0-83f9-740a2ce1cb71\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb"
Oct 30 19:57:52.997099 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:52.997059    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bdc83b4-72fc-42b0-83f9-740a2ce1cb71-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb\" (UID: \"6bdc83b4-72fc-42b0-83f9-740a2ce1cb71\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb"
Oct 30 19:57:52.997265 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:52.997141    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6l5q\" (UniqueName: \"kubernetes.io/projected/6bdc83b4-72fc-42b0-83f9-740a2ce1cb71-kube-api-access-m6l5q\") pod \"payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb\" (UID: \"6bdc83b4-72fc-42b0-83f9-740a2ce1cb71\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb"
Oct 30 19:57:52.997814 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:52.997790    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bdc83b4-72fc-42b0-83f9-740a2ce1cb71-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb\" (UID: \"6bdc83b4-72fc-42b0-83f9-740a2ce1cb71\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb"
Oct 30 19:57:53.079922 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:53.079895    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6l5q\" (UniqueName: \"kubernetes.io/projected/6bdc83b4-72fc-42b0-83f9-740a2ce1cb71-kube-api-access-m6l5q\") pod \"payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb\" (UID: \"6bdc83b4-72fc-42b0-83f9-740a2ce1cb71\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb"
Oct 30 19:57:53.274835 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:53.274807    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp"]
Oct 30 19:57:53.276940 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:57:53.276915    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd4cbac_918f_4517_bdb7_9c4fcecf2401.slice/crio-729935eabc3f5eab930874fdc59402813207d630327ea8a9b36abb512e018f86 WatchSource:0}: Error finding container 729935eabc3f5eab930874fdc59402813207d630327ea8a9b36abb512e018f86: Status 404 returned error can't find the container with id 729935eabc3f5eab930874fdc59402813207d630327ea8a9b36abb512e018f86
Oct 30 19:57:53.355923 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:53.355899    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb"
Oct 30 19:57:53.735403 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:53.735372    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb"]
Oct 30 19:57:53.739322 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:57:53.739287    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bdc83b4_72fc_42b0_83f9_740a2ce1cb71.slice/crio-b7cbdbc452936a0077674e30b226c08e2197bec61f07ae19481f6b429fea1155 WatchSource:0}: Error finding container b7cbdbc452936a0077674e30b226c08e2197bec61f07ae19481f6b429fea1155: Status 404 returned error can't find the container with id b7cbdbc452936a0077674e30b226c08e2197bec61f07ae19481f6b429fea1155
Oct 30 19:57:54.267786 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:54.267749    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb" event={"ID":"6bdc83b4-72fc-42b0-83f9-740a2ce1cb71","Type":"ContainerStarted","Data":"c0e6d5c2c21b5c54b27450229391a7a70639939d1862f2a82cb4ac8d7bc5c733"}
Oct 30 19:57:54.267952 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:54.267798    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb" event={"ID":"6bdc83b4-72fc-42b0-83f9-740a2ce1cb71","Type":"ContainerStarted","Data":"b7cbdbc452936a0077674e30b226c08e2197bec61f07ae19481f6b429fea1155"}
Oct 30 19:57:54.267952 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:54.267835    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb"
Oct 30 19:57:54.269398 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:54.269370    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp" event={"ID":"fbd4cbac-918f-4517-bdb7-9c4fcecf2401","Type":"ContainerStarted","Data":"44bf9565b02e5439fd7ffebcbab406f1c86a902e4f13789625c21bb621d1dbdf"}
Oct 30 19:57:54.269535 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:54.269408    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp"
Oct 30 19:57:54.269535 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:54.269424    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp" event={"ID":"fbd4cbac-918f-4517-bdb7-9c4fcecf2401","Type":"ContainerStarted","Data":"729935eabc3f5eab930874fdc59402813207d630327ea8a9b36abb512e018f86"}
Oct 30 19:57:54.385920 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:54.385871    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb" podStartSLOduration=2.385860536 podStartE2EDuration="2.385860536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:54.383874613 +0000 UTC m=+913.116090003" watchObservedRunningTime="2026-10-30 19:57:54.385860536 +0000 UTC m=+913.118075925"
Oct 30 19:57:54.507350 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:54.507284    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp" podStartSLOduration=2.507265811 podStartE2EDuration="2.507265811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:54.505953319 +0000 UTC m=+913.238168710" watchObservedRunningTime="2026-10-30 19:57:54.507265811 +0000 UTC m=+913.239481202"
Oct 30 19:57:57.986915 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:57.986881    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm"]
Oct 30 19:57:57.990647 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:57.990626    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm"
Oct 30 19:57:58.093172 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:58.093140    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-407632-dockercfg-vmbjf\""
Oct 30 19:57:58.093172 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:58.093151    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-407632\""
Oct 30 19:57:58.095191 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:58.095167    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm"]
Oct 30 19:57:58.144986 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:58.144954    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82"]
Oct 30 19:57:58.147701 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:58.147674    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psqld\" (UniqueName: \"kubernetes.io/projected/f17664d0-97ab-432b-8e09-dbf95dc52b15-kube-api-access-psqld\") pod \"payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm\" (UID: \"f17664d0-97ab-432b-8e09-dbf95dc52b15\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm"
Oct 30 19:57:58.147811 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:58.147725    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f17664d0-97ab-432b-8e09-dbf95dc52b15-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm\" (UID: \"f17664d0-97ab-432b-8e09-dbf95dc52b15\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm"
Oct 30 19:57:58.148443 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:58.148427    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82"
Oct 30 19:57:58.206606 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:58.206576    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82"]
Oct 30 19:57:58.248580 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:58.248523    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bfb13d3-e216-41f2-996d-11541708e890-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-407632-96cf68f6-scr82\" (UID: \"4bfb13d3-e216-41f2-996d-11541708e890\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82"
Oct 30 19:57:58.248702 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:58.248655    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psqld\" (UniqueName: \"kubernetes.io/projected/f17664d0-97ab-432b-8e09-dbf95dc52b15-kube-api-access-psqld\") pod \"payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm\" (UID: \"f17664d0-97ab-432b-8e09-dbf95dc52b15\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm"
Oct 30 19:57:58.248702 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:58.248682    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7xcr\" (UniqueName: \"kubernetes.io/projected/4bfb13d3-e216-41f2-996d-11541708e890-kube-api-access-v7xcr\") pod \"payload-processing-e2e-worker-w5-407632-96cf68f6-scr82\" (UID: \"4bfb13d3-e216-41f2-996d-11541708e890\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82"
Oct 30 19:57:58.248804 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:58.248785    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f17664d0-97ab-432b-8e09-dbf95dc52b15-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm\" (UID: \"f17664d0-97ab-432b-8e09-dbf95dc52b15\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm"
Oct 30 19:57:58.249393 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:58.249368    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f17664d0-97ab-432b-8e09-dbf95dc52b15-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm\" (UID: \"f17664d0-97ab-432b-8e09-dbf95dc52b15\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm"
Oct 30 19:57:58.310316 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:58.310290    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psqld\" (UniqueName: \"kubernetes.io/projected/f17664d0-97ab-432b-8e09-dbf95dc52b15-kube-api-access-psqld\") pod \"payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm\" (UID: \"f17664d0-97ab-432b-8e09-dbf95dc52b15\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm"
Oct 30 19:57:58.350380 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:58.350356    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bfb13d3-e216-41f2-996d-11541708e890-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-407632-96cf68f6-scr82\" (UID: \"4bfb13d3-e216-41f2-996d-11541708e890\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82"
Oct 30 19:57:58.350528 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:58.350510    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7xcr\" (UniqueName: \"kubernetes.io/projected/4bfb13d3-e216-41f2-996d-11541708e890-kube-api-access-v7xcr\") pod \"payload-processing-e2e-worker-w5-407632-96cf68f6-scr82\" (UID: \"4bfb13d3-e216-41f2-996d-11541708e890\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82"
Oct 30 19:57:58.350982 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:58.350951    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bfb13d3-e216-41f2-996d-11541708e890-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-407632-96cf68f6-scr82\" (UID: \"4bfb13d3-e216-41f2-996d-11541708e890\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82"
Oct 30 19:57:58.415544 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:58.415523    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7xcr\" (UniqueName: \"kubernetes.io/projected/4bfb13d3-e216-41f2-996d-11541708e890-kube-api-access-v7xcr\") pod \"payload-processing-e2e-worker-w5-407632-96cf68f6-scr82\" (UID: \"4bfb13d3-e216-41f2-996d-11541708e890\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82"
Oct 30 19:57:58.461470 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:58.461433    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82"
Oct 30 19:57:58.604333 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:58.604304    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm"
Oct 30 19:57:58.839924 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:58.839889    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82"]
Oct 30 19:57:58.844212 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:57:58.844181    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bfb13d3_e216_41f2_996d_11541708e890.slice/crio-68c58e62bc7b443f30f984cbf4d406260026a309508c8c8fa00fe391631a942a WatchSource:0}: Error finding container 68c58e62bc7b443f30f984cbf4d406260026a309508c8c8fa00fe391631a942a: Status 404 returned error can't find the container with id 68c58e62bc7b443f30f984cbf4d406260026a309508c8c8fa00fe391631a942a
Oct 30 19:57:58.979983 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:58.979957    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm"]
Oct 30 19:57:58.981799 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:57:58.981771    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf17664d0_97ab_432b_8e09_dbf95dc52b15.slice/crio-19f3b104606bcffa00b3ed5381fc2516e65440a2271b1147d7fe7e9538cda6fd WatchSource:0}: Error finding container 19f3b104606bcffa00b3ed5381fc2516e65440a2271b1147d7fe7e9538cda6fd: Status 404 returned error can't find the container with id 19f3b104606bcffa00b3ed5381fc2516e65440a2271b1147d7fe7e9538cda6fd
Oct 30 19:57:59.295059 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:59.294970    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82" event={"ID":"4bfb13d3-e216-41f2-996d-11541708e890","Type":"ContainerStarted","Data":"29327240b0ad52a5e4f5030c7582e87efcc50f187c1db4488f2252c76b738775"}
Oct 30 19:57:59.295059 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:59.295030    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82" event={"ID":"4bfb13d3-e216-41f2-996d-11541708e890","Type":"ContainerStarted","Data":"68c58e62bc7b443f30f984cbf4d406260026a309508c8c8fa00fe391631a942a"}
Oct 30 19:57:59.295580 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:59.295111    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82"
Oct 30 19:57:59.296563 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:59.296542    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm" event={"ID":"f17664d0-97ab-432b-8e09-dbf95dc52b15","Type":"ContainerStarted","Data":"83cc183fc372d25dea8c7a9ea816929e623b0981abeeb81907b2005b201ea67d"}
Oct 30 19:57:59.296662 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:59.296577    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm" event={"ID":"f17664d0-97ab-432b-8e09-dbf95dc52b15","Type":"ContainerStarted","Data":"19f3b104606bcffa00b3ed5381fc2516e65440a2271b1147d7fe7e9538cda6fd"}
Oct 30 19:57:59.296662 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:59.296644    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm"
Oct 30 19:57:59.413416 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:57:59.413371    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82" podStartSLOduration=1.41336014 podStartE2EDuration="1.41336014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:59.412128859 +0000 UTC m=+918.144344251" watchObservedRunningTime="2026-10-30 19:57:59.41336014 +0000 UTC m=+918.145575529"
Oct 30 19:58:00.779296 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:00.779230    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm" podStartSLOduration=3.7792165349999998 podStartE2EDuration="3.779216535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:59.531546498 +0000 UTC m=+918.263761889" watchObservedRunningTime="2026-10-30 19:58:00.779216535 +0000 UTC m=+919.511431926"
Oct 30 19:58:00.781007 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:00.780980    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns"]
Oct 30 19:58:00.784860 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:00.784841    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns"
Oct 30 19:58:00.839758 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:00.839634    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-779921\""
Oct 30 19:58:00.874758 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:00.874727    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks89p\" (UniqueName: \"kubernetes.io/projected/58884900-a56f-40e1-8ce3-aea388a61245-kube-api-access-ks89p\") pod \"payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns\" (UID: \"58884900-a56f-40e1-8ce3-aea388a61245\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns"
Oct 30 19:58:00.874957 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:00.874867    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58884900-a56f-40e1-8ce3-aea388a61245-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns\" (UID: \"58884900-a56f-40e1-8ce3-aea388a61245\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns"
Oct 30 19:58:00.888011 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:00.887986    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-779921-dockercfg-kwdgk\""
Oct 30 19:58:00.889750 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:00.889729    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns"]
Oct 30 19:58:00.941353 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:00.941250    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv"]
Oct 30 19:58:00.945542 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:00.945522    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv"
Oct 30 19:58:00.975494 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:00.975470    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv"]
Oct 30 19:58:00.976010 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:00.975989    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks89p\" (UniqueName: \"kubernetes.io/projected/58884900-a56f-40e1-8ce3-aea388a61245-kube-api-access-ks89p\") pod \"payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns\" (UID: \"58884900-a56f-40e1-8ce3-aea388a61245\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns"
Oct 30 19:58:00.976125 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:00.976108    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58884900-a56f-40e1-8ce3-aea388a61245-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns\" (UID: \"58884900-a56f-40e1-8ce3-aea388a61245\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns"
Oct 30 19:58:00.976800 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:00.976779    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58884900-a56f-40e1-8ce3-aea388a61245-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns\" (UID: \"58884900-a56f-40e1-8ce3-aea388a61245\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns"
Oct 30 19:58:01.040040 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:01.039986    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks89p\" (UniqueName: \"kubernetes.io/projected/58884900-a56f-40e1-8ce3-aea388a61245-kube-api-access-ks89p\") pod \"payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns\" (UID: \"58884900-a56f-40e1-8ce3-aea388a61245\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns"
Oct 30 19:58:01.077554 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:01.077533    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d98ef7-ed5f-4f06-b7f8-0ff074f41033-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv\" (UID: \"e0d98ef7-ed5f-4f06-b7f8-0ff074f41033\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv"
Oct 30 19:58:01.077646 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:01.077603    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vrdm\" (UniqueName: \"kubernetes.io/projected/e0d98ef7-ed5f-4f06-b7f8-0ff074f41033-kube-api-access-2vrdm\") pod \"payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv\" (UID: \"e0d98ef7-ed5f-4f06-b7f8-0ff074f41033\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv"
Oct 30 19:58:01.098350 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:01.098326    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns"
Oct 30 19:58:01.179252 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:01.179223    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d98ef7-ed5f-4f06-b7f8-0ff074f41033-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv\" (UID: \"e0d98ef7-ed5f-4f06-b7f8-0ff074f41033\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv"
Oct 30 19:58:01.179420 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:01.179318    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vrdm\" (UniqueName: \"kubernetes.io/projected/e0d98ef7-ed5f-4f06-b7f8-0ff074f41033-kube-api-access-2vrdm\") pod \"payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv\" (UID: \"e0d98ef7-ed5f-4f06-b7f8-0ff074f41033\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv"
Oct 30 19:58:01.180030 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:01.180009    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d98ef7-ed5f-4f06-b7f8-0ff074f41033-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv\" (UID: \"e0d98ef7-ed5f-4f06-b7f8-0ff074f41033\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv"
Oct 30 19:58:01.248943 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:01.248906    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vrdm\" (UniqueName: \"kubernetes.io/projected/e0d98ef7-ed5f-4f06-b7f8-0ff074f41033-kube-api-access-2vrdm\") pod \"payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv\" (UID: \"e0d98ef7-ed5f-4f06-b7f8-0ff074f41033\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv"
Oct 30 19:58:01.263695 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:01.263662    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv"
Oct 30 19:58:01.477707 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:01.477629    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns"]
Oct 30 19:58:01.480557 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:58:01.480516    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58884900_a56f_40e1_8ce3_aea388a61245.slice/crio-eecbd3f82df975120064b5221dbfad13287e0abeda1c080cb4a79f8fefa65efb WatchSource:0}: Error finding container eecbd3f82df975120064b5221dbfad13287e0abeda1c080cb4a79f8fefa65efb: Status 404 returned error can't find the container with id eecbd3f82df975120064b5221dbfad13287e0abeda1c080cb4a79f8fefa65efb
Oct 30 19:58:01.650999 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:01.650926    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv"]
Oct 30 19:58:01.654834 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:58:01.654808    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d98ef7_ed5f_4f06_b7f8_0ff074f41033.slice/crio-d1c7b6146b035906699a4647d0151882cb24a26f0ff65031ad9b5a865fe0fc84 WatchSource:0}: Error finding container d1c7b6146b035906699a4647d0151882cb24a26f0ff65031ad9b5a865fe0fc84: Status 404 returned error can't find the container with id d1c7b6146b035906699a4647d0151882cb24a26f0ff65031ad9b5a865fe0fc84
Oct 30 19:58:02.312630 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:02.312582    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv" event={"ID":"e0d98ef7-ed5f-4f06-b7f8-0ff074f41033","Type":"ContainerStarted","Data":"b2f7158ac1b49f7e5189c0d4f2f97b742369b8db0533d029446313106c15f0b0"}
Oct 30 19:58:02.313042 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:02.312669    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv" event={"ID":"e0d98ef7-ed5f-4f06-b7f8-0ff074f41033","Type":"ContainerStarted","Data":"d1c7b6146b035906699a4647d0151882cb24a26f0ff65031ad9b5a865fe0fc84"}
Oct 30 19:58:02.313042 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:02.312700    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv"
Oct 30 19:58:02.314187 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:02.314163    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns" event={"ID":"58884900-a56f-40e1-8ce3-aea388a61245","Type":"ContainerStarted","Data":"a19c64ec02ca84415ca31e7962bc489858e2308f2a30a6f21319ff19f1c3e0a5"}
Oct 30 19:58:02.314289 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:02.314199    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns" event={"ID":"58884900-a56f-40e1-8ce3-aea388a61245","Type":"ContainerStarted","Data":"eecbd3f82df975120064b5221dbfad13287e0abeda1c080cb4a79f8fefa65efb"}
Oct 30 19:58:02.314289 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:02.314217    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns"
Oct 30 19:58:02.586489 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:02.586421    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns" podStartSLOduration=2.586405748 podStartE2EDuration="2.586405748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:02.584823976 +0000 UTC m=+921.317039366" watchObservedRunningTime="2026-10-30 19:58:02.586405748 +0000 UTC m=+921.318621138"
Oct 30 19:58:02.587891 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:02.587858    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv" podStartSLOduration=2.587851616 podStartE2EDuration="2.587851616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:02.452106234 +0000 UTC m=+921.184321626" watchObservedRunningTime="2026-10-30 19:58:02.587851616 +0000 UTC m=+921.320067005"
Oct 30 19:58:03.778243 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:03.778211    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv"]
Oct 30 19:58:03.786642 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:03.786578    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv"
Oct 30 19:58:03.809971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:03.809943    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv"]
Oct 30 19:58:03.867659 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:03.867633    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-3ccc9f\""
Oct 30 19:58:03.867659 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:03.867636    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-3ccc9f-dockercfg-644jg\""
Oct 30 19:58:03.908639 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:03.908606    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nfpb\" (UniqueName: \"kubernetes.io/projected/a984e6de-54fc-4ec0-83b0-ab295078699b-kube-api-access-7nfpb\") pod \"payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv\" (UID: \"a984e6de-54fc-4ec0-83b0-ab295078699b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv"
Oct 30 19:58:03.908784 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:03.908648    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a984e6de-54fc-4ec0-83b0-ab295078699b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv\" (UID: \"a984e6de-54fc-4ec0-83b0-ab295078699b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv"
Oct 30 19:58:04.010149 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:04.010113    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nfpb\" (UniqueName: \"kubernetes.io/projected/a984e6de-54fc-4ec0-83b0-ab295078699b-kube-api-access-7nfpb\") pod \"payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv\" (UID: \"a984e6de-54fc-4ec0-83b0-ab295078699b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv"
Oct 30 19:58:04.010149 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:04.010152    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a984e6de-54fc-4ec0-83b0-ab295078699b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv\" (UID: \"a984e6de-54fc-4ec0-83b0-ab295078699b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv"
Oct 30 19:58:04.010852 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:04.010832    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a984e6de-54fc-4ec0-83b0-ab295078699b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv\" (UID: \"a984e6de-54fc-4ec0-83b0-ab295078699b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv"
Oct 30 19:58:04.073226 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:04.073201    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nfpb\" (UniqueName: \"kubernetes.io/projected/a984e6de-54fc-4ec0-83b0-ab295078699b-kube-api-access-7nfpb\") pod \"payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv\" (UID: \"a984e6de-54fc-4ec0-83b0-ab295078699b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv"
Oct 30 19:58:04.102021 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:04.101995    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv"
Oct 30 19:58:04.499098 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:04.499076    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv"]
Oct 30 19:58:04.501274 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:58:04.501244    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda984e6de_54fc_4ec0_83b0_ab295078699b.slice/crio-efd5881e89499709f6e1edaea465de58f5f3ccb73105b0309fa4a1548be7f3c5 WatchSource:0}: Error finding container efd5881e89499709f6e1edaea465de58f5f3ccb73105b0309fa4a1548be7f3c5: Status 404 returned error can't find the container with id efd5881e89499709f6e1edaea465de58f5f3ccb73105b0309fa4a1548be7f3c5
Oct 30 19:58:05.276532 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:05.276507    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp"
Oct 30 19:58:05.276942 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:05.276568    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb"
Oct 30 19:58:05.331698 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:05.331667    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv" event={"ID":"a984e6de-54fc-4ec0-83b0-ab295078699b","Type":"ContainerStarted","Data":"5ea23c4035783d7c6ef73fb410cad68711dcfc8f71babfd04ff0a24c814a3020"}
Oct 30 19:58:05.331846 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:05.331722    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv"
Oct 30 19:58:05.331846 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:05.331742    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv" event={"ID":"a984e6de-54fc-4ec0-83b0-ab295078699b","Type":"ContainerStarted","Data":"efd5881e89499709f6e1edaea465de58f5f3ccb73105b0309fa4a1548be7f3c5"}
Oct 30 19:58:05.663621 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:05.663555    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv" podStartSLOduration=2.66353704 podStartE2EDuration="2.66353704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:05.661040402 +0000 UTC m=+924.393255794" watchObservedRunningTime="2026-10-30 19:58:05.66353704 +0000 UTC m=+924.395752430"
Oct 30 19:58:06.617161 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:58:06.617125    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:58:10.311484 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:10.310922    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm"
Oct 30 19:58:10.319517 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:10.319488    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82"
Oct 30 19:58:11.388311 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:11.388277    5958 generic.go:356] "Generic (PLEG): container finished" podID="fde4d62b-bd67-4915-a145-e5511e651c25" containerID="5804a85bb63e293e5e14f36db543f910ccec895a02f22cc2da98da5a2fcc5d16" exitCode=0
Oct 30 19:58:11.388738 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:11.388349    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v" event={"ID":"fde4d62b-bd67-4915-a145-e5511e651c25","Type":"ContainerDied","Data":"5804a85bb63e293e5e14f36db543f910ccec895a02f22cc2da98da5a2fcc5d16"}
Oct 30 19:58:11.388809 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:11.388744    5958 scope.go:122] "RemoveContainer" containerID="5804a85bb63e293e5e14f36db543f910ccec895a02f22cc2da98da5a2fcc5d16"
Oct 30 19:58:11.389945 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:11.389916    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:58:11.389998 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:11.389985    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:58:11.390035 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:11.390016    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:58:11.789528 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:11.789433    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9"]
Oct 30 19:58:11.802188 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:11.802159    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9"
Oct 30 19:58:11.820514 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:11.820488    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9\" (UID: \"8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9"
Oct 30 19:58:11.820647 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:11.820624    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqfkx\" (UniqueName: \"kubernetes.io/projected/8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7-kube-api-access-lqfkx\") pod \"payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9\" (UID: \"8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9"
Oct 30 19:58:11.852034 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:11.852004    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9"]
Oct 30 19:58:11.856154 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:11.856131    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-d67ea9-dockercfg-zx9np\""
Oct 30 19:58:11.856281 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:11.856208    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-d67ea9\""
Oct 30 19:58:11.922161 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:11.922122    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9\" (UID: \"8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9"
Oct 30 19:58:11.922373 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:11.922278    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqfkx\" (UniqueName: \"kubernetes.io/projected/8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7-kube-api-access-lqfkx\") pod \"payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9\" (UID: \"8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9"
Oct 30 19:58:11.923823 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:11.923788    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9\" (UID: \"8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9"
Oct 30 19:58:11.950500 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:11.950464    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7"]
Oct 30 19:58:11.954348 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:11.954331    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7"
Oct 30 19:58:11.984114 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:11.984088    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqfkx\" (UniqueName: \"kubernetes.io/projected/8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7-kube-api-access-lqfkx\") pod \"payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9\" (UID: \"8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9"
Oct 30 19:58:12.019869 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:12.019815    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7"]
Oct 30 19:58:12.025472 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:12.024145    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtdc7\" (UniqueName: \"kubernetes.io/projected/ecb7666c-9fbc-4d05-942b-de443eef8f24-kube-api-access-qtdc7\") pod \"payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7\" (UID: \"ecb7666c-9fbc-4d05-942b-de443eef8f24\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7"
Oct 30 19:58:12.025472 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:12.024332    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb7666c-9fbc-4d05-942b-de443eef8f24-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7\" (UID: \"ecb7666c-9fbc-4d05-942b-de443eef8f24\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7"
Oct 30 19:58:12.115150 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:12.115123    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9"
Oct 30 19:58:12.125610 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:12.125568    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtdc7\" (UniqueName: \"kubernetes.io/projected/ecb7666c-9fbc-4d05-942b-de443eef8f24-kube-api-access-qtdc7\") pod \"payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7\" (UID: \"ecb7666c-9fbc-4d05-942b-de443eef8f24\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7"
Oct 30 19:58:12.125800 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:12.125776    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb7666c-9fbc-4d05-942b-de443eef8f24-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7\" (UID: \"ecb7666c-9fbc-4d05-942b-de443eef8f24\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7"
Oct 30 19:58:12.126735 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:12.126708    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb7666c-9fbc-4d05-942b-de443eef8f24-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7\" (UID: \"ecb7666c-9fbc-4d05-942b-de443eef8f24\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7"
Oct 30 19:58:12.188842 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:12.188806    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtdc7\" (UniqueName: \"kubernetes.io/projected/ecb7666c-9fbc-4d05-942b-de443eef8f24-kube-api-access-qtdc7\") pod \"payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7\" (UID: \"ecb7666c-9fbc-4d05-942b-de443eef8f24\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7"
Oct 30 19:58:12.267676 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:12.267637    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7"
Oct 30 19:58:12.395896 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:12.395818    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v" event={"ID":"fde4d62b-bd67-4915-a145-e5511e651c25","Type":"ContainerStarted","Data":"096bdd50d9cdf384afa596e1c28c74943640d61fdae6fc60613abbf3eb8082a7"}
Oct 30 19:58:12.528043 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:12.528015    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9"]
Oct 30 19:58:12.529357 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:58:12.529331    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8572d9b7_b8ce_4f88_bfd6_7ec0e27e9fa7.slice/crio-9349b99d75e42be2655e0bc9daabfaafb32b0700ca93f674ac66979663044519 WatchSource:0}: Error finding container 9349b99d75e42be2655e0bc9daabfaafb32b0700ca93f674ac66979663044519: Status 404 returned error can't find the container with id 9349b99d75e42be2655e0bc9daabfaafb32b0700ca93f674ac66979663044519
Oct 30 19:58:12.662526 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:12.651139    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7"]
Oct 30 19:58:12.669306 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:58:12.669274    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb7666c_9fbc_4d05_942b_de443eef8f24.slice/crio-28ef11dfa4b3b564b4d6b188cb82aebfdaee6feffdfb53949b03f80bd0efc306 WatchSource:0}: Error finding container 28ef11dfa4b3b564b4d6b188cb82aebfdaee6feffdfb53949b03f80bd0efc306: Status 404 returned error can't find the container with id 28ef11dfa4b3b564b4d6b188cb82aebfdaee6feffdfb53949b03f80bd0efc306
Oct 30 19:58:12.798741 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:12.798702    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:58:12.799677 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:12.799653    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 19:58:12.799804 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:12.799694    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v" podUID="fde4d62b-bd67-4915-a145-e5511e651c25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 19:58:13.322419 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:13.322392    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv"
Oct 30 19:58:13.322597 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:13.322481    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns"
Oct 30 19:58:13.401998 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:13.401966    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7" event={"ID":"ecb7666c-9fbc-4d05-942b-de443eef8f24","Type":"ContainerStarted","Data":"236e59c673a70ccb8fb878c8b08cce59158415b6c08f9b9376f83f7c6552296e"}
Oct 30 19:58:13.402410 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:13.402022    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7" event={"ID":"ecb7666c-9fbc-4d05-942b-de443eef8f24","Type":"ContainerStarted","Data":"28ef11dfa4b3b564b4d6b188cb82aebfdaee6feffdfb53949b03f80bd0efc306"}
Oct 30 19:58:13.402410 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:13.402053    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7"
Oct 30 19:58:13.403504 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:13.403477    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9" event={"ID":"8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7","Type":"ContainerStarted","Data":"78c651d3fe63278b6163ca4afbb6cb89eaa1484d9188f7f3988edcbf28237f3b"}
Oct 30 19:58:13.403646 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:13.403524    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9" event={"ID":"8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7","Type":"ContainerStarted","Data":"9349b99d75e42be2655e0bc9daabfaafb32b0700ca93f674ac66979663044519"}
Oct 30 19:58:13.403717 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:13.403670    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9"
Oct 30 19:58:13.689709 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:13.689605    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9" podStartSLOduration=2.689592085 podStartE2EDuration="2.689592085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:13.688675516 +0000 UTC m=+932.420890908" watchObservedRunningTime="2026-10-30 19:58:13.689592085 +0000 UTC m=+932.421807474"
Oct 30 19:58:13.798981 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:13.798950    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 19:58:13.799138 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:13.799000    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v" podUID="fde4d62b-bd67-4915-a145-e5511e651c25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 19:58:13.821138 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:13.821090    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7" podStartSLOduration=2.821073841 podStartE2EDuration="2.821073841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:13.819781987 +0000 UTC m=+932.551997377" watchObservedRunningTime="2026-10-30 19:58:13.821073841 +0000 UTC m=+932.553289235"
Oct 30 19:58:14.216503 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:14.216440    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:58:14.799117 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:14.799082    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 19:58:14.799664 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:14.799141    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v" podUID="fde4d62b-bd67-4915-a145-e5511e651c25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 19:58:15.799421 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:15.799394    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 19:58:15.799810 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:15.799442    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v" podUID="fde4d62b-bd67-4915-a145-e5511e651c25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 19:58:16.361393 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:16.361367    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv"
Oct 30 19:58:16.799568 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:16.799487    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 19:58:16.799568 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:16.799544    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v" podUID="fde4d62b-bd67-4915-a145-e5511e651c25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 19:58:17.803314 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:17.803287    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:58:17.804321 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:17.804301    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 19:58:24.411078 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:24.411047    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9"
Oct 30 19:58:24.411570 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:24.411119    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7"
Oct 30 19:58:28.510390 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:28.510298    5958 generic.go:356] "Generic (PLEG): container finished" podID="6bc60e1a-391d-43b0-8b8e-2f6f33c234be" containerID="e05dc617cb1c861aedefcb6d4afe1c26b2a0a634e86006c15482f836f64e38a8" exitCode=0
Oct 30 19:58:28.510390 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:28.510343    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6" event={"ID":"6bc60e1a-391d-43b0-8b8e-2f6f33c234be","Type":"ContainerDied","Data":"e05dc617cb1c861aedefcb6d4afe1c26b2a0a634e86006c15482f836f64e38a8"}
Oct 30 19:58:28.510821 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:28.510733    5958 scope.go:122] "RemoveContainer" containerID="e05dc617cb1c861aedefcb6d4afe1c26b2a0a634e86006c15482f836f64e38a8"
Oct 30 19:58:28.512046 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:28.512020    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:58:28.512105 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:28.512083    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:58:28.512141 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:28.512113    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:58:29.111131 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:29.111097    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z"]
Oct 30 19:58:29.114085 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:29.114063    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z"
Oct 30 19:58:29.168593 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:29.168555    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:58:29.168834 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:29.168706    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-bebb42-serving-cert\""
Oct 30 19:58:29.168834 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:29.168743    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:58:29.168834 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:29.168823    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xtx52\""
Oct 30 19:58:29.216902 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:29.216881    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 19:58:29.219388 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:29.219361    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z"]
Oct 30 19:58:29.261374 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:29.261323    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js2ws\" (UniqueName: \"kubernetes.io/projected/2836bfd6-8a39-4194-8ac7-723940c07a2e-kube-api-access-js2ws\") pod \"maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z\" (UID: \"2836bfd6-8a39-4194-8ac7-723940c07a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z"
Oct 30 19:58:29.261584 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:29.261404    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2836bfd6-8a39-4194-8ac7-723940c07a2e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z\" (UID: \"2836bfd6-8a39-4194-8ac7-723940c07a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z"
Oct 30 19:58:29.261584 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:29.261522    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2836bfd6-8a39-4194-8ac7-723940c07a2e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z\" (UID: \"2836bfd6-8a39-4194-8ac7-723940c07a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z"
Oct 30 19:58:29.261584 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:29.261561    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2836bfd6-8a39-4194-8ac7-723940c07a2e-metrics-certs\") pod \"maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z\" (UID: \"2836bfd6-8a39-4194-8ac7-723940c07a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z"
Oct 30 19:58:29.362919 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:29.362808    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2836bfd6-8a39-4194-8ac7-723940c07a2e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z\" (UID: \"2836bfd6-8a39-4194-8ac7-723940c07a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z"
Oct 30 19:58:29.362919 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:29.362914    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2836bfd6-8a39-4194-8ac7-723940c07a2e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z\" (UID: \"2836bfd6-8a39-4194-8ac7-723940c07a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z"
Oct 30 19:58:29.363157 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:29.362970    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2836bfd6-8a39-4194-8ac7-723940c07a2e-metrics-certs\") pod \"maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z\" (UID: \"2836bfd6-8a39-4194-8ac7-723940c07a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z"
Oct 30 19:58:29.363157 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:58:29.363008    5958 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-serving-cert: secret "maas-api-e2e-worker-w4-bebb42-serving-cert" not found
Oct 30 19:58:29.363157 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:58:29.363089    5958 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2836bfd6-8a39-4194-8ac7-723940c07a2e-maas-api-tls podName:2836bfd6-8a39-4194-8ac7-723940c07a2e nodeName:}" failed. No retries permitted until 2026-10-30 19:58:29.863067534 +0000 UTC m=+948.595282925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2836bfd6-8a39-4194-8ac7-723940c07a2e-maas-api-tls") pod "maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z" (UID: "2836bfd6-8a39-4194-8ac7-723940c07a2e") : secret "maas-api-e2e-worker-w4-bebb42-serving-cert" not found
Oct 30 19:58:29.363157 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:29.363150    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js2ws\" (UniqueName: \"kubernetes.io/projected/2836bfd6-8a39-4194-8ac7-723940c07a2e-kube-api-access-js2ws\") pod \"maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z\" (UID: \"2836bfd6-8a39-4194-8ac7-723940c07a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z"
Oct 30 19:58:29.365118 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:29.364274    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2836bfd6-8a39-4194-8ac7-723940c07a2e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z\" (UID: \"2836bfd6-8a39-4194-8ac7-723940c07a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z"
Oct 30 19:58:29.366965 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:29.366935    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2836bfd6-8a39-4194-8ac7-723940c07a2e-metrics-certs\") pod \"maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z\" (UID: \"2836bfd6-8a39-4194-8ac7-723940c07a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z"
Oct 30 19:58:29.427703 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:29.427670    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js2ws\" (UniqueName: \"kubernetes.io/projected/2836bfd6-8a39-4194-8ac7-723940c07a2e-kube-api-access-js2ws\") pod \"maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z\" (UID: \"2836bfd6-8a39-4194-8ac7-723940c07a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z"
Oct 30 19:58:29.518035 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:29.517995    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6" event={"ID":"6bc60e1a-391d-43b0-8b8e-2f6f33c234be","Type":"ContainerStarted","Data":"31ab70206f3a82a88027c08ead3041b13125165887f999631d71abfdc35bdfa6"}
Oct 30 19:58:29.869775 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:29.869728    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:58:29.870241 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:29.870199    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2836bfd6-8a39-4194-8ac7-723940c07a2e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z\" (UID: \"2836bfd6-8a39-4194-8ac7-723940c07a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z"
Oct 30 19:58:29.871060 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:29.871037    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:58:29.871167 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:29.871091    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6" podUID="6bc60e1a-391d-43b0-8b8e-2f6f33c234be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 19:58:29.872983 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:29.872952    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2836bfd6-8a39-4194-8ac7-723940c07a2e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z\" (UID: \"2836bfd6-8a39-4194-8ac7-723940c07a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z"
Oct 30 19:58:30.026321 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:30.026294    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z"
Oct 30 19:58:30.222081 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:30.222005    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:58:30.434629 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:30.426716    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z"]
Oct 30 19:58:30.434629 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:30.430338    5958 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:58:30.523932 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:30.523867    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z" event={"ID":"2836bfd6-8a39-4194-8ac7-723940c07a2e","Type":"ContainerStarted","Data":"6609203cd3efff75c5885fb43bde2115df76b2936c9ddd9fc991f112a216b604"}
Oct 30 19:58:30.869695 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:30.869657    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:58:30.869892 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:30.869730    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6" podUID="6bc60e1a-391d-43b0-8b8e-2f6f33c234be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 19:58:31.881971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:31.881930    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:58:31.882612 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:31.882565    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6" podUID="6bc60e1a-391d-43b0-8b8e-2f6f33c234be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 19:58:32.884475 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:32.870836    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:58:32.884475 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:32.870900    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6" podUID="6bc60e1a-391d-43b0-8b8e-2f6f33c234be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 19:58:33.870334 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:33.869857    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:58:33.870334 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:33.869925    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6" podUID="6bc60e1a-391d-43b0-8b8e-2f6f33c234be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 19:58:34.886804 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:34.886767    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:58:34.889368 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:34.889184    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 19:58:36.613846 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:58:36.613811    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:58:37.673604 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:37.673570    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z" event={"ID":"2836bfd6-8a39-4194-8ac7-723940c07a2e","Type":"ContainerStarted","Data":"00c42a3bab2be68efc4789676a87de8972fdc5e4ada9878fc454048dd9e13415"}
Oct 30 19:58:37.673995 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:37.673656    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z"
Oct 30 19:58:37.794977 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:37.794923    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z" podStartSLOduration=2.133801777 podStartE2EDuration="8.794910325s" totalImagesPullingTime="6.661108548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:37.793160055 +0000 UTC m=+956.525375446" watchObservedRunningTime="2026-10-30 19:58:37.794910325 +0000 UTC m=+956.527125714"
Oct 30 19:58:40.700675 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:40.700639    5958 generic.go:356] "Generic (PLEG): container finished" podID="56ec387e-8e34-472e-b8f5-032a57c517e9" containerID="a706cb79798838937a181d0a6787ccfddc79c46487fd13d6073c0ea1910bba0c" exitCode=0
Oct 30 19:58:40.701081 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:40.700730    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw" event={"ID":"56ec387e-8e34-472e-b8f5-032a57c517e9","Type":"ContainerDied","Data":"a706cb79798838937a181d0a6787ccfddc79c46487fd13d6073c0ea1910bba0c"}
Oct 30 19:58:40.701246 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:40.701228    5958 scope.go:122] "RemoveContainer" containerID="a706cb79798838937a181d0a6787ccfddc79c46487fd13d6073c0ea1910bba0c"
Oct 30 19:58:40.702739 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:40.702705    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:58:40.702873 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:40.702772    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:58:40.702873 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:40.702806    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:58:41.707966 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:41.707922    5958 generic.go:356] "Generic (PLEG): container finished" podID="590c5cad-401a-49be-a4b1-79892dc39d5d" containerID="44b5342dca4dbe6d31592852da154980e2c82c0088a84e20ba8a998573a55bcc" exitCode=0
Oct 30 19:58:41.708428 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:41.707997    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg" event={"ID":"590c5cad-401a-49be-a4b1-79892dc39d5d","Type":"ContainerDied","Data":"44b5342dca4dbe6d31592852da154980e2c82c0088a84e20ba8a998573a55bcc"}
Oct 30 19:58:41.708428 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:41.708420    5958 scope.go:122] "RemoveContainer" containerID="44b5342dca4dbe6d31592852da154980e2c82c0088a84e20ba8a998573a55bcc"
Oct 30 19:58:41.709913 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:41.709883    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:58:41.710039 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:41.709958    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:58:41.710039 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:41.709996    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 19:58:41.710125 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:41.710098    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw" event={"ID":"56ec387e-8e34-472e-b8f5-032a57c517e9","Type":"ContainerStarted","Data":"7036812e0caf371d6f0c015abbfc0c9efd36138d7ed80f62d938d577c2ce2b03"}
Oct 30 19:58:42.497053 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:42.497022    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:58:42.498097 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:42.498075    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 19:58:42.498148 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:42.498122    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw" podUID="56ec387e-8e34-472e-b8f5-032a57c517e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 19:58:42.717913 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:42.717878    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg" event={"ID":"590c5cad-401a-49be-a4b1-79892dc39d5d","Type":"ContainerStarted","Data":"0376c7d0aae1c7f91cda1b39b9eb4e5dfb133fdaec604d26f6fc482ac89a8f0d"}
Oct 30 19:58:43.130553 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:43.130520    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:58:43.131896 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:43.131866    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 19:58:43.132008 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:43.131918    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg" podUID="590c5cad-401a-49be-a4b1-79892dc39d5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 19:58:43.497018 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:43.496924    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 19:58:43.497018 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:43.496985    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw" podUID="56ec387e-8e34-472e-b8f5-032a57c517e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 19:58:43.692169 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:43.692129    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z"
Oct 30 19:58:44.131445 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:44.131413    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 19:58:44.131854 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:44.131487    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg" podUID="590c5cad-401a-49be-a4b1-79892dc39d5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 19:58:44.216282 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:44.216230    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:58:44.496323 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:44.496237    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 19:58:44.496323 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:44.496287    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw" podUID="56ec387e-8e34-472e-b8f5-032a57c517e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 19:58:45.131325 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:45.131287    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 19:58:45.131537 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:45.131344    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg" podUID="590c5cad-401a-49be-a4b1-79892dc39d5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 19:58:45.221928 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:45.221897    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:58:45.496943 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:45.496860    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 19:58:45.496943 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:45.496915    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw" podUID="56ec387e-8e34-472e-b8f5-032a57c517e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 19:58:46.130638 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:46.130597    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 19:58:46.130843 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:46.130660    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg" podUID="590c5cad-401a-49be-a4b1-79892dc39d5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 19:58:46.497020 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:46.496933    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 19:58:46.497020 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:46.496984    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw" podUID="56ec387e-8e34-472e-b8f5-032a57c517e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 19:58:47.131134 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:47.131104    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 19:58:47.131302 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:47.131162    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg" podUID="590c5cad-401a-49be-a4b1-79892dc39d5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 19:58:47.500754 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:47.500688    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:58:47.501625 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:47.501608    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 19:58:48.135186 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:48.135156    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:58:48.136471 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:58:48.136436    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 19:59:06.606141 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:59:06.606106    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:59:08.253043 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.253007    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"]
Oct 30 19:59:08.256987 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.256967    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 19:59:08.291059 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.291029    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-tls-certs\") pod \"simulator-w0-779921-kserve-58f5755476-grz6k\" (UID: \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\") " pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 19:59:08.291059 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.291061    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-tmp-dir\") pod \"simulator-w0-779921-kserve-58f5755476-grz6k\" (UID: \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\") " pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 19:59:08.291291 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.291089    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rz9r\" (UniqueName: \"kubernetes.io/projected/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-kube-api-access-5rz9r\") pod \"simulator-w0-779921-kserve-58f5755476-grz6k\" (UID: \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\") " pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 19:59:08.291291 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.291223    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-model-cache\") pod \"simulator-w0-779921-kserve-58f5755476-grz6k\" (UID: \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\") " pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 19:59:08.291291 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.291253    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-dshm\") pod \"simulator-w0-779921-kserve-58f5755476-grz6k\" (UID: \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\") " pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 19:59:08.291483 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.291316    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-home\") pod \"simulator-w0-779921-kserve-58f5755476-grz6k\" (UID: \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\") " pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 19:59:08.359549 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.359520    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-779921\"/\"kube-root-ca.crt\""
Oct 30 19:59:08.359713 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.359636    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-779921\"/\"openshift-service-ca.crt\""
Oct 30 19:59:08.359713 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.359668    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-779921\"/\"default-dockercfg-wdcjk\""
Oct 30 19:59:08.359713 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.359705    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-779921\"/\"simulator-w0-779921-kserve-self-signed-certs\""
Oct 30 19:59:08.361711 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.361694    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"]
Oct 30 19:59:08.393210 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.393188    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-home\") pod \"simulator-w0-779921-kserve-58f5755476-grz6k\" (UID: \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\") " pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 19:59:08.393912 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.393885    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-tls-certs\") pod \"simulator-w0-779921-kserve-58f5755476-grz6k\" (UID: \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\") " pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 19:59:08.394070 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.394054    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-tmp-dir\") pod \"simulator-w0-779921-kserve-58f5755476-grz6k\" (UID: \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\") " pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 19:59:08.394192 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.394176    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rz9r\" (UniqueName: \"kubernetes.io/projected/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-kube-api-access-5rz9r\") pod \"simulator-w0-779921-kserve-58f5755476-grz6k\" (UID: \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\") " pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 19:59:08.395247 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.395185    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-tmp-dir\") pod \"simulator-w0-779921-kserve-58f5755476-grz6k\" (UID: \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\") " pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 19:59:08.395247 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.393745    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-home\") pod \"simulator-w0-779921-kserve-58f5755476-grz6k\" (UID: \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\") " pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 19:59:08.395523 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.395500    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-model-cache\") pod \"simulator-w0-779921-kserve-58f5755476-grz6k\" (UID: \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\") " pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 19:59:08.402396 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.395964    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-model-cache\") pod \"simulator-w0-779921-kserve-58f5755476-grz6k\" (UID: \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\") " pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 19:59:08.402633 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.402611    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-dshm\") pod \"simulator-w0-779921-kserve-58f5755476-grz6k\" (UID: \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\") " pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 19:59:08.405597 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.405551    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-dshm\") pod \"simulator-w0-779921-kserve-58f5755476-grz6k\" (UID: \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\") " pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 19:59:08.405710 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.405627    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-tls-certs\") pod \"simulator-w0-779921-kserve-58f5755476-grz6k\" (UID: \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\") " pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 19:59:08.458232 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.458207    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rz9r\" (UniqueName: \"kubernetes.io/projected/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-kube-api-access-5rz9r\") pod \"simulator-w0-779921-kserve-58f5755476-grz6k\" (UID: \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\") " pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 19:59:08.569150 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.569124    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 19:59:08.945159 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:08.945133    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"]
Oct 30 19:59:09.907013 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:09.906975    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" event={"ID":"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b","Type":"ContainerStarted","Data":"8cc6ee3c0a5a70aab4aaeba2a4a86f5545e39cd420ff5909029fcd255e24dc94"}
Oct 30 19:59:09.907013 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:09.907015    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" event={"ID":"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b","Type":"ContainerStarted","Data":"9d16046da8b75d87322cab986147d6964f75696d123aec5868b88bdd6ac60d45"}
Oct 30 19:59:10.042951 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:10.042900    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" podStartSLOduration=2.042887546 podStartE2EDuration="2.042887546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:10.040985462 +0000 UTC m=+988.773200851" watchObservedRunningTime="2026-10-30 19:59:10.042887546 +0000 UTC m=+988.775102936"
Oct 30 19:59:10.570066 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:10.570035    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 19:59:18.569869 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:18.569830    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 19:59:18.581955 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:18.581935    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 19:59:18.966767 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:18.966689    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 19:59:23.416383 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.416300    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"]
Oct 30 19:59:23.420293 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.420272    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 19:59:23.523075 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.523029    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f84d85\"/\"openshift-service-ca.crt\""
Oct 30 19:59:23.523245 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.523075    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f84d85\"/\"simulator-w3-f84d85-kserve-self-signed-certs\""
Oct 30 19:59:23.523245 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.523193    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f84d85\"/\"default-dockercfg-8kngq\""
Oct 30 19:59:23.526442 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.526422    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"]
Oct 30 19:59:23.554473 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.554427    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-tls-certs\") pod \"simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk\" (UID: \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\") " pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 19:59:23.554564 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.554494    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-dshm\") pod \"simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk\" (UID: \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\") " pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 19:59:23.554609 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.554556    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bdpt\" (UniqueName: \"kubernetes.io/projected/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-kube-api-access-4bdpt\") pod \"simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk\" (UID: \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\") " pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 19:59:23.554609 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.554603    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-model-cache\") pod \"simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk\" (UID: \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\") " pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 19:59:23.554672 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.554628    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-home\") pod \"simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk\" (UID: \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\") " pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 19:59:23.554755 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.554740    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-tmp-dir\") pod \"simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk\" (UID: \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\") " pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 19:59:23.573827 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.573770    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f84d85\"/\"kube-root-ca.crt\""
Oct 30 19:59:23.655914 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.655886    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-tmp-dir\") pod \"simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk\" (UID: \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\") " pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 19:59:23.656044 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.655964    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-tls-certs\") pod \"simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk\" (UID: \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\") " pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 19:59:23.656044 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.655988    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-dshm\") pod \"simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk\" (UID: \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\") " pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 19:59:23.656044 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.656006    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bdpt\" (UniqueName: \"kubernetes.io/projected/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-kube-api-access-4bdpt\") pod \"simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk\" (UID: \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\") " pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 19:59:23.656044 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.656038    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-model-cache\") pod \"simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk\" (UID: \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\") " pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 19:59:23.656271 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.656062    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-home\") pod \"simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk\" (UID: \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\") " pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 19:59:23.656363 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.656340    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-tmp-dir\") pod \"simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk\" (UID: \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\") " pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 19:59:23.656444 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.656421    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-model-cache\") pod \"simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk\" (UID: \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\") " pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 19:59:23.656512 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.656484    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-home\") pod \"simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk\" (UID: \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\") " pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 19:59:23.658191 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.658172    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-dshm\") pod \"simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk\" (UID: \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\") " pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 19:59:23.658562 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.658544    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-tls-certs\") pod \"simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk\" (UID: \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\") " pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 19:59:23.717062 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.717000    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bdpt\" (UniqueName: \"kubernetes.io/projected/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-kube-api-access-4bdpt\") pod \"simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk\" (UID: \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\") " pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 19:59:23.732309 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:23.732287    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 19:59:24.105790 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:24.105761    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"]
Oct 30 19:59:24.107896 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:59:24.107870    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1929aa5_fcd4_4609_8cbc_68afb34bdadc.slice/crio-7aaff9259a95f34e21404f7dd3e3fd6c3039b1ff23ce327d19c3ffcbbd3a7033 WatchSource:0}: Error finding container 7aaff9259a95f34e21404f7dd3e3fd6c3039b1ff23ce327d19c3ffcbbd3a7033: Status 404 returned error can't find the container with id 7aaff9259a95f34e21404f7dd3e3fd6c3039b1ff23ce327d19c3ffcbbd3a7033
Oct 30 19:59:24.990524 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:24.990484    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" event={"ID":"d1929aa5-fcd4-4609-8cbc-68afb34bdadc","Type":"ContainerStarted","Data":"c9e70d47d0fa20467a1fcaff683f5fd6422dd47ec24f1d979b23542a31354833"}
Oct 30 19:59:24.990971 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:24.990540    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" event={"ID":"d1929aa5-fcd4-4609-8cbc-68afb34bdadc","Type":"ContainerStarted","Data":"7aaff9259a95f34e21404f7dd3e3fd6c3039b1ff23ce327d19c3ffcbbd3a7033"}
Oct 30 19:59:25.112035 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:25.111987    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podStartSLOduration=2.111971838 podStartE2EDuration="2.111971838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:25.110309337 +0000 UTC m=+1003.842524727" watchObservedRunningTime="2026-10-30 19:59:25.111971838 +0000 UTC m=+1003.844187229"
Oct 30 19:59:25.733176 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:25.733146    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 19:59:33.733506 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:33.733471    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 19:59:33.745311 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:33.745288    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 19:59:34.049838 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:34.049763    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 19:59:36.606040 ip-10-0-142-240 kubenswrapper[5958]: E1030 19:59:36.606005    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 19:59:51.507265 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.507229    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"]
Oct 30 19:59:51.512197 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.512177    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 19:59:51.618048 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.618013    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f75272\"/\"simulator-w6-f75272-kserve-self-signed-certs\""
Oct 30 19:59:51.618212 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.618110    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f75272\"/\"default-dockercfg-qdntr\""
Oct 30 19:59:51.618212 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.618161    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-f75272\"/\"kube-root-ca.crt\""
Oct 30 19:59:51.618309 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.618207    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-f75272\"/\"openshift-service-ca.crt\""
Oct 30 19:59:51.619683 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.619662    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-tmp-dir\") pod \"simulator-w6-f75272-kserve-685889b95d-tp9gh\" (UID: \"55e64b7b-c7ad-474e-beeb-a8b221deb328\") " pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 19:59:51.619752 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.619734    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4ppw\" (UniqueName: \"kubernetes.io/projected/55e64b7b-c7ad-474e-beeb-a8b221deb328-kube-api-access-t4ppw\") pod \"simulator-w6-f75272-kserve-685889b95d-tp9gh\" (UID: \"55e64b7b-c7ad-474e-beeb-a8b221deb328\") " pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 19:59:51.619873 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.619856    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-home\") pod \"simulator-w6-f75272-kserve-685889b95d-tp9gh\" (UID: \"55e64b7b-c7ad-474e-beeb-a8b221deb328\") " pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 19:59:51.619950 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.619936    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-model-cache\") pod \"simulator-w6-f75272-kserve-685889b95d-tp9gh\" (UID: \"55e64b7b-c7ad-474e-beeb-a8b221deb328\") " pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 19:59:51.620055 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.620040    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-dshm\") pod \"simulator-w6-f75272-kserve-685889b95d-tp9gh\" (UID: \"55e64b7b-c7ad-474e-beeb-a8b221deb328\") " pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 19:59:51.620115 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.620053    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"]
Oct 30 19:59:51.620115 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.620084    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e64b7b-c7ad-474e-beeb-a8b221deb328-tls-certs\") pod \"simulator-w6-f75272-kserve-685889b95d-tp9gh\" (UID: \"55e64b7b-c7ad-474e-beeb-a8b221deb328\") " pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 19:59:51.720730 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.720687    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-model-cache\") pod \"simulator-w6-f75272-kserve-685889b95d-tp9gh\" (UID: \"55e64b7b-c7ad-474e-beeb-a8b221deb328\") " pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 19:59:51.720915 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.720778    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-dshm\") pod \"simulator-w6-f75272-kserve-685889b95d-tp9gh\" (UID: \"55e64b7b-c7ad-474e-beeb-a8b221deb328\") " pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 19:59:51.720915 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.720819    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e64b7b-c7ad-474e-beeb-a8b221deb328-tls-certs\") pod \"simulator-w6-f75272-kserve-685889b95d-tp9gh\" (UID: \"55e64b7b-c7ad-474e-beeb-a8b221deb328\") " pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 19:59:51.720915 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.720867    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-tmp-dir\") pod \"simulator-w6-f75272-kserve-685889b95d-tp9gh\" (UID: \"55e64b7b-c7ad-474e-beeb-a8b221deb328\") " pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 19:59:51.721088 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.720930    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4ppw\" (UniqueName: \"kubernetes.io/projected/55e64b7b-c7ad-474e-beeb-a8b221deb328-kube-api-access-t4ppw\") pod \"simulator-w6-f75272-kserve-685889b95d-tp9gh\" (UID: \"55e64b7b-c7ad-474e-beeb-a8b221deb328\") " pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 19:59:51.721088 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.720986    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-home\") pod \"simulator-w6-f75272-kserve-685889b95d-tp9gh\" (UID: \"55e64b7b-c7ad-474e-beeb-a8b221deb328\") " pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 19:59:51.721194 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.721138    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-model-cache\") pod \"simulator-w6-f75272-kserve-685889b95d-tp9gh\" (UID: \"55e64b7b-c7ad-474e-beeb-a8b221deb328\") " pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 19:59:51.721267 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.721239    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-tmp-dir\") pod \"simulator-w6-f75272-kserve-685889b95d-tp9gh\" (UID: \"55e64b7b-c7ad-474e-beeb-a8b221deb328\") " pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 19:59:51.721305 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.721259    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-home\") pod \"simulator-w6-f75272-kserve-685889b95d-tp9gh\" (UID: \"55e64b7b-c7ad-474e-beeb-a8b221deb328\") " pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 19:59:51.723217 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.723195    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-dshm\") pod \"simulator-w6-f75272-kserve-685889b95d-tp9gh\" (UID: \"55e64b7b-c7ad-474e-beeb-a8b221deb328\") " pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 19:59:51.723464 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.723423    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e64b7b-c7ad-474e-beeb-a8b221deb328-tls-certs\") pod \"simulator-w6-f75272-kserve-685889b95d-tp9gh\" (UID: \"55e64b7b-c7ad-474e-beeb-a8b221deb328\") " pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 19:59:51.786920 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.786858    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4ppw\" (UniqueName: \"kubernetes.io/projected/55e64b7b-c7ad-474e-beeb-a8b221deb328-kube-api-access-t4ppw\") pod \"simulator-w6-f75272-kserve-685889b95d-tp9gh\" (UID: \"55e64b7b-c7ad-474e-beeb-a8b221deb328\") " pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 19:59:51.823256 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:51.823222    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 19:59:52.197829 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:52.197790    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"]
Oct 30 19:59:52.200974 ip-10-0-142-240 kubenswrapper[5958]: W1030 19:59:52.200947    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55e64b7b_c7ad_474e_beeb_a8b221deb328.slice/crio-dbf08008b0914cf5f2374f69b4cb4890ea8a8538230b5abcdcfaf39651611f9e WatchSource:0}: Error finding container dbf08008b0914cf5f2374f69b4cb4890ea8a8538230b5abcdcfaf39651611f9e: Status 404 returned error can't find the container with id dbf08008b0914cf5f2374f69b4cb4890ea8a8538230b5abcdcfaf39651611f9e
Oct 30 19:59:53.140960 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:53.140925    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" event={"ID":"55e64b7b-c7ad-474e-beeb-a8b221deb328","Type":"ContainerStarted","Data":"7fbcc9a64dd557a0f9d630a704a13c5a9fd6e6bb44db9b2eb3bd317c7e83f347"}
Oct 30 19:59:53.140960 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:53.140969    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" event={"ID":"55e64b7b-c7ad-474e-beeb-a8b221deb328","Type":"ContainerStarted","Data":"dbf08008b0914cf5f2374f69b4cb4890ea8a8538230b5abcdcfaf39651611f9e"}
Oct 30 19:59:53.259409 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:53.259359    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podStartSLOduration=2.25934688 podStartE2EDuration="2.25934688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:53.258526578 +0000 UTC m=+1031.990741969" watchObservedRunningTime="2026-10-30 19:59:53.25934688 +0000 UTC m=+1031.991562282"
Oct 30 19:59:53.824245 ip-10-0-142-240 kubenswrapper[5958]: I1030 19:59:53.824212    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 20:00:00.199710 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.199681    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv"]
Oct 30 20:00:00.203957 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.203935    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv"]
Oct 30 20:00:00.204093 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.204045    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv"
Oct 30 20:00:00.207389 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.207369    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv"
Oct 30 20:00:00.226728 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.226704    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp"]
Oct 30 20:00:00.244850 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.244814    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp"
Oct 30 20:00:00.256654 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.256635    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:00:00.306312 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.306285    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv"]
Oct 30 20:00:00.307361 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.307336    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3a0190-8e00-4099-851c-d0c6302899fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-q4vhp\" (UID: \"ec3a0190-8e00-4099-851c-d0c6302899fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp"
Oct 30 20:00:00.307591 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.307563    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f01965-ced2-4a8f-976b-0ac5ed37e6f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv\" (UID: \"78f01965-ced2-4a8f-976b-0ac5ed37e6f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv"
Oct 30 20:00:00.307696 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.307633    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn986\" (UniqueName: \"kubernetes.io/projected/84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a-kube-api-access-sn986\") pod \"maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv\" (UID: \"84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv"
Oct 30 20:00:00.307787 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.307763    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcrgs\" (UniqueName: \"kubernetes.io/projected/78f01965-ced2-4a8f-976b-0ac5ed37e6f3-kube-api-access-lcrgs\") pod \"maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv\" (UID: \"78f01965-ced2-4a8f-976b-0ac5ed37e6f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv"
Oct 30 20:00:00.307866 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.307848    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv\" (UID: \"84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv"
Oct 30 20:00:00.307910 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.307894    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xg8z\" (UniqueName: \"kubernetes.io/projected/ec3a0190-8e00-4099-851c-d0c6302899fe-kube-api-access-5xg8z\") pod \"maas-api-key-cleanup-29889840-q4vhp\" (UID: \"ec3a0190-8e00-4099-851c-d0c6302899fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp"
Oct 30 20:00:00.308463 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.308433    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv"]
Oct 30 20:00:00.351766 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.351736    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp"]
Oct 30 20:00:00.408556 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.408532    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv\" (UID: \"84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv"
Oct 30 20:00:00.408700 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.408573    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xg8z\" (UniqueName: \"kubernetes.io/projected/ec3a0190-8e00-4099-851c-d0c6302899fe-kube-api-access-5xg8z\") pod \"maas-api-key-cleanup-29889840-q4vhp\" (UID: \"ec3a0190-8e00-4099-851c-d0c6302899fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp"
Oct 30 20:00:00.408790 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.408768    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3a0190-8e00-4099-851c-d0c6302899fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-q4vhp\" (UID: \"ec3a0190-8e00-4099-851c-d0c6302899fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp"
Oct 30 20:00:00.408851 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.408828    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f01965-ced2-4a8f-976b-0ac5ed37e6f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv\" (UID: \"78f01965-ced2-4a8f-976b-0ac5ed37e6f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv"
Oct 30 20:00:00.408943 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.408871    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn986\" (UniqueName: \"kubernetes.io/projected/84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a-kube-api-access-sn986\") pod \"maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv\" (UID: \"84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv"
Oct 30 20:00:00.408943 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.408920    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcrgs\" (UniqueName: \"kubernetes.io/projected/78f01965-ced2-4a8f-976b-0ac5ed37e6f3-kube-api-access-lcrgs\") pod \"maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv\" (UID: \"78f01965-ced2-4a8f-976b-0ac5ed37e6f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv"
Oct 30 20:00:00.409271 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.409250    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv\" (UID: \"84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv"
Oct 30 20:00:00.409356 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.409339    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3a0190-8e00-4099-851c-d0c6302899fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-q4vhp\" (UID: \"ec3a0190-8e00-4099-851c-d0c6302899fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp"
Oct 30 20:00:00.409491 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.409438    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f01965-ced2-4a8f-976b-0ac5ed37e6f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv\" (UID: \"78f01965-ced2-4a8f-976b-0ac5ed37e6f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv"
Oct 30 20:00:00.469983 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.469929    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xg8z\" (UniqueName: \"kubernetes.io/projected/ec3a0190-8e00-4099-851c-d0c6302899fe-kube-api-access-5xg8z\") pod \"maas-api-key-cleanup-29889840-q4vhp\" (UID: \"ec3a0190-8e00-4099-851c-d0c6302899fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp"
Oct 30 20:00:00.470070 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.470006    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcrgs\" (UniqueName: \"kubernetes.io/projected/78f01965-ced2-4a8f-976b-0ac5ed37e6f3-kube-api-access-lcrgs\") pod \"maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv\" (UID: \"78f01965-ced2-4a8f-976b-0ac5ed37e6f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv"
Oct 30 20:00:00.470070 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.470005    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn986\" (UniqueName: \"kubernetes.io/projected/84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a-kube-api-access-sn986\") pod \"maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv\" (UID: \"84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv"
Oct 30 20:00:00.518495 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.518476    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv"
Oct 30 20:00:00.525366 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.525346    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv"
Oct 30 20:00:00.557195 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.557166    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp"
Oct 30 20:00:00.895324 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.895299    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv"]
Oct 30 20:00:00.897121 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:00:00.897097    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a37f5b_c4f5_4f08_b0ea_420bc56d9e2a.slice/crio-aaad620a9e6af2440b27ab3e355f6ac8a2b74be74b1ff4cb94c5795bac695232 WatchSource:0}: Error finding container aaad620a9e6af2440b27ab3e355f6ac8a2b74be74b1ff4cb94c5795bac695232: Status 404 returned error can't find the container with id aaad620a9e6af2440b27ab3e355f6ac8a2b74be74b1ff4cb94c5795bac695232
Oct 30 20:00:00.946316 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.946294    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv"]
Oct 30 20:00:00.947519 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:00:00.947500    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f01965_ced2_4a8f_976b_0ac5ed37e6f3.slice/crio-912e641e42bb78c3ebb401a16ca1c68dd8b535bfeec67d8c472dc5c412bb9fdb WatchSource:0}: Error finding container 912e641e42bb78c3ebb401a16ca1c68dd8b535bfeec67d8c472dc5c412bb9fdb: Status 404 returned error can't find the container with id 912e641e42bb78c3ebb401a16ca1c68dd8b535bfeec67d8c472dc5c412bb9fdb
Oct 30 20:00:00.959697 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:00.959677    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp"]
Oct 30 20:00:00.960980 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:00:00.960962    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec3a0190_8e00_4099_851c_d0c6302899fe.slice/crio-7a28a78d8a3632319665cde52c5230bfc6465281a1bba3636338c95a1750c04c WatchSource:0}: Error finding container 7a28a78d8a3632319665cde52c5230bfc6465281a1bba3636338c95a1750c04c: Status 404 returned error can't find the container with id 7a28a78d8a3632319665cde52c5230bfc6465281a1bba3636338c95a1750c04c
Oct 30 20:00:01.194473 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.194381    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv" event={"ID":"84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a","Type":"ContainerStarted","Data":"aaad620a9e6af2440b27ab3e355f6ac8a2b74be74b1ff4cb94c5795bac695232"}
Oct 30 20:00:01.195662 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.195639    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp" event={"ID":"ec3a0190-8e00-4099-851c-d0c6302899fe","Type":"ContainerStarted","Data":"7a28a78d8a3632319665cde52c5230bfc6465281a1bba3636338c95a1750c04c"}
Oct 30 20:00:01.196898 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.196873    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv" event={"ID":"78f01965-ced2-4a8f-976b-0ac5ed37e6f3","Type":"ContainerStarted","Data":"912e641e42bb78c3ebb401a16ca1c68dd8b535bfeec67d8c472dc5c412bb9fdb"}
Oct 30 20:00:01.353559 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.353533    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"]
Oct 30 20:00:01.358740 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.358723    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:00:01.416997 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.416972    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bebb42\"/\"default-dockercfg-h8x5s\""
Oct 30 20:00:01.416997 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.416972    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bebb42\"/\"openshift-service-ca.crt\""
Oct 30 20:00:01.417162 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.417008    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bebb42\"/\"kube-root-ca.crt\""
Oct 30 20:00:01.418510 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.418491    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-tmp-dir\") pod \"simulator-w4-bebb42-kserve-b759cb9d-h5xpg\" (UID: \"7804ed32-f5d5-41b7-856f-ceebcad9a306\") " pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:00:01.418570 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.418535    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-home\") pod \"simulator-w4-bebb42-kserve-b759cb9d-h5xpg\" (UID: \"7804ed32-f5d5-41b7-856f-ceebcad9a306\") " pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:00:01.418570 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.418555    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-model-cache\") pod \"simulator-w4-bebb42-kserve-b759cb9d-h5xpg\" (UID: \"7804ed32-f5d5-41b7-856f-ceebcad9a306\") " pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:00:01.418729 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.418709    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-dshm\") pod \"simulator-w4-bebb42-kserve-b759cb9d-h5xpg\" (UID: \"7804ed32-f5d5-41b7-856f-ceebcad9a306\") " pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:00:01.418796 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.418776    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k5bg\" (UniqueName: \"kubernetes.io/projected/7804ed32-f5d5-41b7-856f-ceebcad9a306-kube-api-access-4k5bg\") pod \"simulator-w4-bebb42-kserve-b759cb9d-h5xpg\" (UID: \"7804ed32-f5d5-41b7-856f-ceebcad9a306\") " pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:00:01.418927 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.418908    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7804ed32-f5d5-41b7-856f-ceebcad9a306-tls-certs\") pod \"simulator-w4-bebb42-kserve-b759cb9d-h5xpg\" (UID: \"7804ed32-f5d5-41b7-856f-ceebcad9a306\") " pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:00:01.456672 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.456615    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bebb42\"/\"simulator-w4-bebb42-kserve-self-signed-certs\""
Oct 30 20:00:01.467029 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.467006    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"]
Oct 30 20:00:01.520062 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.520025    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7804ed32-f5d5-41b7-856f-ceebcad9a306-tls-certs\") pod \"simulator-w4-bebb42-kserve-b759cb9d-h5xpg\" (UID: \"7804ed32-f5d5-41b7-856f-ceebcad9a306\") " pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:00:01.520232 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.520082    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-tmp-dir\") pod \"simulator-w4-bebb42-kserve-b759cb9d-h5xpg\" (UID: \"7804ed32-f5d5-41b7-856f-ceebcad9a306\") " pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:00:01.520232 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.520125    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-home\") pod \"simulator-w4-bebb42-kserve-b759cb9d-h5xpg\" (UID: \"7804ed32-f5d5-41b7-856f-ceebcad9a306\") " pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:00:01.520232 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.520160    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-model-cache\") pod \"simulator-w4-bebb42-kserve-b759cb9d-h5xpg\" (UID: \"7804ed32-f5d5-41b7-856f-ceebcad9a306\") " pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:00:01.520232 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.520227    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-dshm\") pod \"simulator-w4-bebb42-kserve-b759cb9d-h5xpg\" (UID: \"7804ed32-f5d5-41b7-856f-ceebcad9a306\") " pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:00:01.520407 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.520249    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k5bg\" (UniqueName: \"kubernetes.io/projected/7804ed32-f5d5-41b7-856f-ceebcad9a306-kube-api-access-4k5bg\") pod \"simulator-w4-bebb42-kserve-b759cb9d-h5xpg\" (UID: \"7804ed32-f5d5-41b7-856f-ceebcad9a306\") " pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:00:01.520651 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.520622    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-tmp-dir\") pod \"simulator-w4-bebb42-kserve-b759cb9d-h5xpg\" (UID: \"7804ed32-f5d5-41b7-856f-ceebcad9a306\") " pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:00:01.520765 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.520695    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-home\") pod \"simulator-w4-bebb42-kserve-b759cb9d-h5xpg\" (UID: \"7804ed32-f5d5-41b7-856f-ceebcad9a306\") " pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:00:01.520851 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.520829    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-model-cache\") pod \"simulator-w4-bebb42-kserve-b759cb9d-h5xpg\" (UID: \"7804ed32-f5d5-41b7-856f-ceebcad9a306\") " pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:00:01.522608 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.522579    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-dshm\") pod \"simulator-w4-bebb42-kserve-b759cb9d-h5xpg\" (UID: \"7804ed32-f5d5-41b7-856f-ceebcad9a306\") " pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:00:01.522732 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.522718    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7804ed32-f5d5-41b7-856f-ceebcad9a306-tls-certs\") pod \"simulator-w4-bebb42-kserve-b759cb9d-h5xpg\" (UID: \"7804ed32-f5d5-41b7-856f-ceebcad9a306\") " pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:00:01.582679 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.582654    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k5bg\" (UniqueName: \"kubernetes.io/projected/7804ed32-f5d5-41b7-856f-ceebcad9a306-kube-api-access-4k5bg\") pod \"simulator-w4-bebb42-kserve-b759cb9d-h5xpg\" (UID: \"7804ed32-f5d5-41b7-856f-ceebcad9a306\") " pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:00:01.672340 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.672311    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:00:01.825343 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.823687    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 20:00:01.840035 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:01.840004    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 20:00:02.047012 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:02.046964    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"]
Oct 30 20:00:02.049716 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:00:02.049686    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7804ed32_f5d5_41b7_856f_ceebcad9a306.slice/crio-ffaf9dd5c8e8e8aa8b4b47524cd926b2218f2488dfc007ed955136618558453a WatchSource:0}: Error finding container ffaf9dd5c8e8e8aa8b4b47524cd926b2218f2488dfc007ed955136618558453a: Status 404 returned error can't find the container with id ffaf9dd5c8e8e8aa8b4b47524cd926b2218f2488dfc007ed955136618558453a
Oct 30 20:00:02.204935 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:02.204895    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" event={"ID":"7804ed32-f5d5-41b7-856f-ceebcad9a306","Type":"ContainerStarted","Data":"1cd8d31e8b7c1ce7d603a88ec2f9ac2bb420c8e93beb825787f17adb98ba0074"}
Oct 30 20:00:02.205100 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:02.204947    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" event={"ID":"7804ed32-f5d5-41b7-856f-ceebcad9a306","Type":"ContainerStarted","Data":"ffaf9dd5c8e8e8aa8b4b47524cd926b2218f2488dfc007ed955136618558453a"}
Oct 30 20:00:02.216007 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:02.215977    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 20:00:02.325273 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:02.325210    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podStartSLOduration=1.325193867 podStartE2EDuration="1.325193867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.323714304 +0000 UTC m=+1041.055929694" watchObservedRunningTime="2026-10-30 20:00:02.325193867 +0000 UTC m=+1041.057409258"
Oct 30 20:00:02.672746 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:02.672677    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:00:06.237260 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:06.237203    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp" event={"ID":"ec3a0190-8e00-4099-851c-d0c6302899fe","Type":"ContainerStarted","Data":"18dc45ad9a036ced3de0f111432edb319e97ab5ad4c179a25b2587bfaa37fa2c"}
Oct 30 20:00:06.240882 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:06.240429    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv" event={"ID":"78f01965-ced2-4a8f-976b-0ac5ed37e6f3","Type":"ContainerStarted","Data":"5da339b3fb0d64be382bc7a25fddbaffc7d0999bda16a163454705c065958118"}
Oct 30 20:00:06.242999 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:06.242970    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv" event={"ID":"84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a","Type":"ContainerStarted","Data":"b1111880006699a198302375cd85b46e92e0064e4c786678a640813119ab108c"}
Oct 30 20:00:06.361184 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:06.360246    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp" podStartSLOduration=1.807985447 podStartE2EDuration="6.360226783s" totalImagesPullingTime="4.552241336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:06.358742124 +0000 UTC m=+1045.090957515" watchObservedRunningTime="2026-10-30 20:00:06.360226783 +0000 UTC m=+1045.092442174"
Oct 30 20:00:06.480132 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:06.480087    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv" podStartSLOduration=1.9057066329999999 podStartE2EDuration="6.480068948s" totalImagesPullingTime="4.574362315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:06.475721767 +0000 UTC m=+1045.207937155" watchObservedRunningTime="2026-10-30 20:00:06.480068948 +0000 UTC m=+1045.212284338"
Oct 30 20:00:06.593669 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:06.593601    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv" podStartSLOduration=2.126182295 podStartE2EDuration="6.593581802s" totalImagesPullingTime="4.467399507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:06.591724214 +0000 UTC m=+1045.323939605" watchObservedRunningTime="2026-10-30 20:00:06.593581802 +0000 UTC m=+1045.325797218"
Oct 30 20:00:06.608259 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:00:06.608227    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:00:07.249848 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:07.249808    5958 generic.go:356] "Generic (PLEG): container finished" podID="ec3a0190-8e00-4099-851c-d0c6302899fe" containerID="18dc45ad9a036ced3de0f111432edb319e97ab5ad4c179a25b2587bfaa37fa2c" exitCode=60
Oct 30 20:00:07.250269 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:07.249895    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp" event={"ID":"ec3a0190-8e00-4099-851c-d0c6302899fe","Type":"ContainerDied","Data":"18dc45ad9a036ced3de0f111432edb319e97ab5ad4c179a25b2587bfaa37fa2c"}
Oct 30 20:00:07.250269 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:07.250116    5958 scope.go:122] "RemoveContainer" containerID="18dc45ad9a036ced3de0f111432edb319e97ab5ad4c179a25b2587bfaa37fa2c"
Oct 30 20:00:07.251728 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:07.251703    5958 generic.go:356] "Generic (PLEG): container finished" podID="78f01965-ced2-4a8f-976b-0ac5ed37e6f3" containerID="5da339b3fb0d64be382bc7a25fddbaffc7d0999bda16a163454705c065958118" exitCode=60
Oct 30 20:00:07.251856 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:07.251839    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv" event={"ID":"78f01965-ced2-4a8f-976b-0ac5ed37e6f3","Type":"ContainerDied","Data":"5da339b3fb0d64be382bc7a25fddbaffc7d0999bda16a163454705c065958118"}
Oct 30 20:00:07.252055 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:07.252033    5958 scope.go:122] "RemoveContainer" containerID="5da339b3fb0d64be382bc7a25fddbaffc7d0999bda16a163454705c065958118"
Oct 30 20:00:07.253619 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:07.253596    5958 generic.go:356] "Generic (PLEG): container finished" podID="84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a" containerID="b1111880006699a198302375cd85b46e92e0064e4c786678a640813119ab108c" exitCode=60
Oct 30 20:00:07.253718 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:07.253693    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv" event={"ID":"84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a","Type":"ContainerDied","Data":"b1111880006699a198302375cd85b46e92e0064e4c786678a640813119ab108c"}
Oct 30 20:00:07.253890 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:07.253872    5958 scope.go:122] "RemoveContainer" containerID="b1111880006699a198302375cd85b46e92e0064e4c786678a640813119ab108c"
Oct 30 20:00:08.263053 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:08.263020    5958 generic.go:356] "Generic (PLEG): container finished" podID="84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a" containerID="213cf186af209e345685e5ea5707bad7b69061c8120dc0fa04c9e90cd8c36fad" exitCode=60
Oct 30 20:00:08.263434 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:08.263106    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv" event={"ID":"84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a","Type":"ContainerDied","Data":"213cf186af209e345685e5ea5707bad7b69061c8120dc0fa04c9e90cd8c36fad"}
Oct 30 20:00:08.263434 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:08.263173    5958 scope.go:122] "RemoveContainer" containerID="b1111880006699a198302375cd85b46e92e0064e4c786678a640813119ab108c"
Oct 30 20:00:08.263434 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:08.263398    5958 scope.go:122] "RemoveContainer" containerID="213cf186af209e345685e5ea5707bad7b69061c8120dc0fa04c9e90cd8c36fad"
Oct 30 20:00:08.263733 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:00:08.263707    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv_odh-ai-gateway-infra(84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv" podUID="84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a"
Oct 30 20:00:08.266182 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:08.266159    5958 generic.go:356] "Generic (PLEG): container finished" podID="ec3a0190-8e00-4099-851c-d0c6302899fe" containerID="fd677a2444f02e651e68974c4a7b672f5c9657dd5958d00e1e67a4d6455f693a" exitCode=60
Oct 30 20:00:08.266271 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:08.266246    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp" event={"ID":"ec3a0190-8e00-4099-851c-d0c6302899fe","Type":"ContainerDied","Data":"fd677a2444f02e651e68974c4a7b672f5c9657dd5958d00e1e67a4d6455f693a"}
Oct 30 20:00:08.266529 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:08.266498    5958 scope.go:122] "RemoveContainer" containerID="fd677a2444f02e651e68974c4a7b672f5c9657dd5958d00e1e67a4d6455f693a"
Oct 30 20:00:08.266737 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:00:08.266715    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-q4vhp_odh-ai-gateway-infra(ec3a0190-8e00-4099-851c-d0c6302899fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp" podUID="ec3a0190-8e00-4099-851c-d0c6302899fe"
Oct 30 20:00:08.268555 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:08.268531    5958 generic.go:356] "Generic (PLEG): container finished" podID="78f01965-ced2-4a8f-976b-0ac5ed37e6f3" containerID="9bbdb0440dd7dd263ff41d585b7ae5304fd7ae1c24401be7d6f41bc960dc1569" exitCode=60
Oct 30 20:00:08.268640 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:08.268616    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv" event={"ID":"78f01965-ced2-4a8f-976b-0ac5ed37e6f3","Type":"ContainerDied","Data":"9bbdb0440dd7dd263ff41d585b7ae5304fd7ae1c24401be7d6f41bc960dc1569"}
Oct 30 20:00:08.268799 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:08.268783    5958 scope.go:122] "RemoveContainer" containerID="9bbdb0440dd7dd263ff41d585b7ae5304fd7ae1c24401be7d6f41bc960dc1569"
Oct 30 20:00:08.268988 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:00:08.268971    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv_odh-ai-gateway-infra(78f01965-ced2-4a8f-976b-0ac5ed37e6f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv" podUID="78f01965-ced2-4a8f-976b-0ac5ed37e6f3"
Oct 30 20:00:08.316196 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:08.316178    5958 scope.go:122] "RemoveContainer" containerID="18dc45ad9a036ced3de0f111432edb319e97ab5ad4c179a25b2587bfaa37fa2c"
Oct 30 20:00:08.328431 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:08.328411    5958 scope.go:122] "RemoveContainer" containerID="5da339b3fb0d64be382bc7a25fddbaffc7d0999bda16a163454705c065958118"
Oct 30 20:00:09.275369 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:09.275326    5958 scope.go:122] "RemoveContainer" containerID="fd677a2444f02e651e68974c4a7b672f5c9657dd5958d00e1e67a4d6455f693a"
Oct 30 20:00:09.275821 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:00:09.275605    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-q4vhp_odh-ai-gateway-infra(ec3a0190-8e00-4099-851c-d0c6302899fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp" podUID="ec3a0190-8e00-4099-851c-d0c6302899fe"
Oct 30 20:00:09.276853 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:09.276834    5958 scope.go:122] "RemoveContainer" containerID="9bbdb0440dd7dd263ff41d585b7ae5304fd7ae1c24401be7d6f41bc960dc1569"
Oct 30 20:00:09.277048 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:00:09.277029    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv_odh-ai-gateway-infra(78f01965-ced2-4a8f-976b-0ac5ed37e6f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv" podUID="78f01965-ced2-4a8f-976b-0ac5ed37e6f3"
Oct 30 20:00:09.278296 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:09.278281    5958 scope.go:122] "RemoveContainer" containerID="213cf186af209e345685e5ea5707bad7b69061c8120dc0fa04c9e90cd8c36fad"
Oct 30 20:00:09.278445 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:00:09.278427    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv_odh-ai-gateway-infra(84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv" podUID="84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a"
Oct 30 20:00:11.673117 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:11.673088    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:00:11.685531 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:11.685510    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:00:12.314139 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:12.314107    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:00:19.391466 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:19.391421    5958 scope.go:122] "RemoveContainer" containerID="213cf186af209e345685e5ea5707bad7b69061c8120dc0fa04c9e90cd8c36fad"
Oct 30 20:00:20.351434 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:20.351402    5958 generic.go:356] "Generic (PLEG): container finished" podID="84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a" containerID="7e4766e1d31039258c36efb9d1bdc4637c53a3a2d840b55c1bdf47a6fc055dd7" exitCode=60
Oct 30 20:00:20.351623 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:20.351479    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv" event={"ID":"84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a","Type":"ContainerDied","Data":"7e4766e1d31039258c36efb9d1bdc4637c53a3a2d840b55c1bdf47a6fc055dd7"}
Oct 30 20:00:20.351623 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:20.351542    5958 scope.go:122] "RemoveContainer" containerID="213cf186af209e345685e5ea5707bad7b69061c8120dc0fa04c9e90cd8c36fad"
Oct 30 20:00:20.351910 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:20.351884    5958 scope.go:122] "RemoveContainer" containerID="7e4766e1d31039258c36efb9d1bdc4637c53a3a2d840b55c1bdf47a6fc055dd7"
Oct 30 20:00:20.352183 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:00:20.352143    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv_odh-ai-gateway-infra(84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv" podUID="84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a"
Oct 30 20:00:21.394191 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:21.394161    5958 scope.go:122] "RemoveContainer" containerID="fd677a2444f02e651e68974c4a7b672f5c9657dd5958d00e1e67a4d6455f693a"
Oct 30 20:00:21.394584 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:21.394421    5958 scope.go:122] "RemoveContainer" containerID="9bbdb0440dd7dd263ff41d585b7ae5304fd7ae1c24401be7d6f41bc960dc1569"
Oct 30 20:00:21.478151 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:21.478118    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv"]
Oct 30 20:00:21.614920 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:21.614890    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv"
Oct 30 20:00:21.648161 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:21.648094    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a-maas-api-service-ca\") pod \"84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a\" (UID: \"84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a\") "
Oct 30 20:00:21.648292 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:21.648219    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a-kube-api-access-sn986\" (UniqueName: \"kubernetes.io/projected/84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a-kube-api-access-sn986\") pod \"84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a\" (UID: \"84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a\") "
Oct 30 20:00:21.648431 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:21.648408    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a-maas-api-service-ca" pod "84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a" (UID: "84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:21.648973 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:21.648742    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a-maas-api-service-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:21.650773 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:21.650749    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a-kube-api-access-sn986" pod "84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a" (UID: "84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a"). InnerVolumeSpecName "kube-api-access-sn986". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:21.749527 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:21.749502    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn986\" (UniqueName: \"kubernetes.io/projected/84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a-kube-api-access-sn986\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:22.375129 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:22.375096    5958 generic.go:356] "Generic (PLEG): container finished" podID="ec3a0190-8e00-4099-851c-d0c6302899fe" containerID="3a37dd21b7495defa5b3d4491b5d1680a6971d2ee6fa9e1d61197b0662eadbf6" exitCode=60
Oct 30 20:00:22.375269 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:22.375140    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp" event={"ID":"ec3a0190-8e00-4099-851c-d0c6302899fe","Type":"ContainerDied","Data":"3a37dd21b7495defa5b3d4491b5d1680a6971d2ee6fa9e1d61197b0662eadbf6"}
Oct 30 20:00:22.375269 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:22.375229    5958 scope.go:122] "RemoveContainer" containerID="fd677a2444f02e651e68974c4a7b672f5c9657dd5958d00e1e67a4d6455f693a"
Oct 30 20:00:22.375679 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:22.375658    5958 scope.go:122] "RemoveContainer" containerID="3a37dd21b7495defa5b3d4491b5d1680a6971d2ee6fa9e1d61197b0662eadbf6"
Oct 30 20:00:22.375950 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:00:22.375928    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-q4vhp_odh-ai-gateway-infra(ec3a0190-8e00-4099-851c-d0c6302899fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp" podUID="ec3a0190-8e00-4099-851c-d0c6302899fe"
Oct 30 20:00:22.377461 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:22.377427    5958 generic.go:356] "Generic (PLEG): container finished" podID="78f01965-ced2-4a8f-976b-0ac5ed37e6f3" containerID="305676f5ae4710d9073186eab1399bb8a9e79cb32a82da604dc4d03036f5c122" exitCode=60
Oct 30 20:00:22.377539 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:22.377486    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv" event={"ID":"78f01965-ced2-4a8f-976b-0ac5ed37e6f3","Type":"ContainerDied","Data":"305676f5ae4710d9073186eab1399bb8a9e79cb32a82da604dc4d03036f5c122"}
Oct 30 20:00:22.377855 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:22.377837    5958 scope.go:122] "RemoveContainer" containerID="305676f5ae4710d9073186eab1399bb8a9e79cb32a82da604dc4d03036f5c122"
Oct 30 20:00:22.378149 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:00:22.378132    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv_odh-ai-gateway-infra(78f01965-ced2-4a8f-976b-0ac5ed37e6f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv" podUID="78f01965-ced2-4a8f-976b-0ac5ed37e6f3"
Oct 30 20:00:22.378882 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:22.378869    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv"
Oct 30 20:00:22.378928 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:22.378901    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv" event={"ID":"84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a","Type":"ContainerDied","Data":"aaad620a9e6af2440b27ab3e355f6ac8a2b74be74b1ff4cb94c5795bac695232"}
Oct 30 20:00:22.387608 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:22.387586    5958 scope.go:122] "RemoveContainer" containerID="9bbdb0440dd7dd263ff41d585b7ae5304fd7ae1c24401be7d6f41bc960dc1569"
Oct 30 20:00:22.400646 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:22.400534    5958 scope.go:122] "RemoveContainer" containerID="7e4766e1d31039258c36efb9d1bdc4637c53a3a2d840b55c1bdf47a6fc055dd7"
Oct 30 20:00:22.723819 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:22.723746    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv"]
Oct 30 20:00:22.727890 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:22.727859    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-407632-29889840-fqfdv"]
Oct 30 20:00:23.397643 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:23.397605    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a" path="/var/lib/kubelet/pods/84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a/volumes"
Oct 30 20:00:23.512524 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:23.512491    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp"]
Oct 30 20:00:23.625315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:23.625284    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv"]
Oct 30 20:00:23.651252 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:23.651188    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp"
Oct 30 20:00:23.672154 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:23.672128    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec3a0190-8e00-4099-851c-d0c6302899fe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3a0190-8e00-4099-851c-d0c6302899fe-maas-api-service-ca\") pod \"ec3a0190-8e00-4099-851c-d0c6302899fe\" (UID: \"ec3a0190-8e00-4099-851c-d0c6302899fe\") "
Oct 30 20:00:23.672281 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:23.672198    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec3a0190-8e00-4099-851c-d0c6302899fe-kube-api-access-5xg8z\" (UniqueName: \"kubernetes.io/projected/ec3a0190-8e00-4099-851c-d0c6302899fe-kube-api-access-5xg8z\") pod \"ec3a0190-8e00-4099-851c-d0c6302899fe\" (UID: \"ec3a0190-8e00-4099-851c-d0c6302899fe\") "
Oct 30 20:00:23.672494 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:23.672443    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec3a0190-8e00-4099-851c-d0c6302899fe-maas-api-service-ca" pod "ec3a0190-8e00-4099-851c-d0c6302899fe" (UID: "ec3a0190-8e00-4099-851c-d0c6302899fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:23.672703 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:23.672682    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3a0190-8e00-4099-851c-d0c6302899fe-maas-api-service-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:23.674513 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:23.674355    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec3a0190-8e00-4099-851c-d0c6302899fe-kube-api-access-5xg8z" pod "ec3a0190-8e00-4099-851c-d0c6302899fe" (UID: "ec3a0190-8e00-4099-851c-d0c6302899fe"). InnerVolumeSpecName "kube-api-access-5xg8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:23.773749 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:23.773721    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xg8z\" (UniqueName: \"kubernetes.io/projected/ec3a0190-8e00-4099-851c-d0c6302899fe-kube-api-access-5xg8z\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:23.780840 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:23.780818    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv"
Oct 30 20:00:23.874422 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:23.874398    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78f01965-ced2-4a8f-976b-0ac5ed37e6f3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f01965-ced2-4a8f-976b-0ac5ed37e6f3-maas-api-service-ca\") pod \"78f01965-ced2-4a8f-976b-0ac5ed37e6f3\" (UID: \"78f01965-ced2-4a8f-976b-0ac5ed37e6f3\") "
Oct 30 20:00:23.874563 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:23.874443    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78f01965-ced2-4a8f-976b-0ac5ed37e6f3-kube-api-access-lcrgs\" (UniqueName: \"kubernetes.io/projected/78f01965-ced2-4a8f-976b-0ac5ed37e6f3-kube-api-access-lcrgs\") pod \"78f01965-ced2-4a8f-976b-0ac5ed37e6f3\" (UID: \"78f01965-ced2-4a8f-976b-0ac5ed37e6f3\") "
Oct 30 20:00:23.874806 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:23.874783    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78f01965-ced2-4a8f-976b-0ac5ed37e6f3-maas-api-service-ca" pod "78f01965-ced2-4a8f-976b-0ac5ed37e6f3" (UID: "78f01965-ced2-4a8f-976b-0ac5ed37e6f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:23.876574 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:23.876554    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78f01965-ced2-4a8f-976b-0ac5ed37e6f3-kube-api-access-lcrgs" pod "78f01965-ced2-4a8f-976b-0ac5ed37e6f3" (UID: "78f01965-ced2-4a8f-976b-0ac5ed37e6f3"). InnerVolumeSpecName "kube-api-access-lcrgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:23.975779 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:23.975720    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f01965-ced2-4a8f-976b-0ac5ed37e6f3-maas-api-service-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:23.975779 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:23.975745    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcrgs\" (UniqueName: \"kubernetes.io/projected/78f01965-ced2-4a8f-976b-0ac5ed37e6f3-kube-api-access-lcrgs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:24.399466 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:24.399407    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp" event={"ID":"ec3a0190-8e00-4099-851c-d0c6302899fe","Type":"ContainerDied","Data":"7a28a78d8a3632319665cde52c5230bfc6465281a1bba3636338c95a1750c04c"}
Oct 30 20:00:24.399625 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:24.399423    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp"
Oct 30 20:00:24.399625 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:24.399496    5958 scope.go:122] "RemoveContainer" containerID="3a37dd21b7495defa5b3d4491b5d1680a6971d2ee6fa9e1d61197b0662eadbf6"
Oct 30 20:00:24.401794 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:24.401776    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv"
Oct 30 20:00:24.401913 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:24.401810    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv" event={"ID":"78f01965-ced2-4a8f-976b-0ac5ed37e6f3","Type":"ContainerDied","Data":"912e641e42bb78c3ebb401a16ca1c68dd8b535bfeec67d8c472dc5c412bb9fdb"}
Oct 30 20:00:24.410163 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:24.410144    5958 scope.go:122] "RemoveContainer" containerID="305676f5ae4710d9073186eab1399bb8a9e79cb32a82da604dc4d03036f5c122"
Oct 30 20:00:24.567718 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:24.567692    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv"]
Oct 30 20:00:24.574172 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:24.574149    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bebb42-29889840-7bhfv"]
Oct 30 20:00:24.686142 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:24.686085    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp"]
Oct 30 20:00:24.695745 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:24.695725    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-q4vhp"]
Oct 30 20:00:25.398983 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:25.398953    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78f01965-ced2-4a8f-976b-0ac5ed37e6f3" path="/var/lib/kubelet/pods/78f01965-ced2-4a8f-976b-0ac5ed37e6f3/volumes"
Oct 30 20:00:25.399344 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:00:25.399330    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec3a0190-8e00-4099-851c-d0c6302899fe" path="/var/lib/kubelet/pods/ec3a0190-8e00-4099-851c-d0c6302899fe/volumes"
Oct 30 20:00:36.604719 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:00:36.604681    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:01:06.605340 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:01:06.605266    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:01:28.879198 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.879159    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e3a22a6-tf5nn"]
Oct 30 20:01:28.879758 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.879743    5958 cpu_manager.go:414] "containerMap: removing container" podUID="78f01965-ced2-4a8f-976b-0ac5ed37e6f3" containerName="cleanup"
Oct 30 20:01:28.879800 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.879760    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78f01965-ced2-4a8f-976b-0ac5ed37e6f3" containerName="cleanup"
Oct 30 20:01:28.879800 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.879770    5958 cpu_manager.go:414] "containerMap: removing container" podUID="78f01965-ced2-4a8f-976b-0ac5ed37e6f3" containerName="cleanup"
Oct 30 20:01:28.879800 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.879775    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78f01965-ced2-4a8f-976b-0ac5ed37e6f3" containerName="cleanup"
Oct 30 20:01:28.879800 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.879789    5958 cpu_manager.go:414] "containerMap: removing container" podUID="84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a" containerName="cleanup"
Oct 30 20:01:28.879800 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.879795    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a" containerName="cleanup"
Oct 30 20:01:28.879946 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.879804    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ec3a0190-8e00-4099-851c-d0c6302899fe" containerName="cleanup"
Oct 30 20:01:28.879946 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.879812    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec3a0190-8e00-4099-851c-d0c6302899fe" containerName="cleanup"
Oct 30 20:01:28.879946 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.879841    5958 cpu_manager.go:414] "containerMap: removing container" podUID="84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a" containerName="cleanup"
Oct 30 20:01:28.879946 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.879845    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a" containerName="cleanup"
Oct 30 20:01:28.879946 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.879863    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ec3a0190-8e00-4099-851c-d0c6302899fe" containerName="cleanup"
Oct 30 20:01:28.879946 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.879868    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec3a0190-8e00-4099-851c-d0c6302899fe" containerName="cleanup"
Oct 30 20:01:28.880117 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.879957    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a" containerName="cleanup"
Oct 30 20:01:28.880117 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.879970    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a" containerName="cleanup"
Oct 30 20:01:28.880117 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.879979    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec3a0190-8e00-4099-851c-d0c6302899fe" containerName="cleanup"
Oct 30 20:01:28.880117 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.879987    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="78f01965-ced2-4a8f-976b-0ac5ed37e6f3" containerName="cleanup"
Oct 30 20:01:28.880117 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.879993    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="78f01965-ced2-4a8f-976b-0ac5ed37e6f3" containerName="cleanup"
Oct 30 20:01:28.880117 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.879999    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a" containerName="cleanup"
Oct 30 20:01:28.880117 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.880004    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec3a0190-8e00-4099-851c-d0c6302899fe" containerName="cleanup"
Oct 30 20:01:28.883178 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.883163    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a22a6-tf5nn"
Oct 30 20:01:28.964801 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.964767    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2225fad5-f918-4bc2-9b8d-c35c84cf570f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e3a22a6-tf5nn\" (UID: \"2225fad5-f918-4bc2-9b8d-c35c84cf570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a22a6-tf5nn"
Oct 30 20:01:28.964801 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.964804    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p2pv\" (UniqueName: \"kubernetes.io/projected/2225fad5-f918-4bc2-9b8d-c35c84cf570f-kube-api-access-5p2pv\") pod \"maas-api-revoke-sub-1e3a22a6-tf5nn\" (UID: \"2225fad5-f918-4bc2-9b8d-c35c84cf570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a22a6-tf5nn"
Oct 30 20:01:28.965000 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.964893    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2225fad5-f918-4bc2-9b8d-c35c84cf570f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e3a22a6-tf5nn\" (UID: \"2225fad5-f918-4bc2-9b8d-c35c84cf570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a22a6-tf5nn"
Oct 30 20:01:28.987395 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.987370    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:01:28.987395 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.987398    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:01:28.989412 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:28.989392    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e3a22a6-tf5nn"]
Oct 30 20:01:29.066019 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:29.065979    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2225fad5-f918-4bc2-9b8d-c35c84cf570f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e3a22a6-tf5nn\" (UID: \"2225fad5-f918-4bc2-9b8d-c35c84cf570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a22a6-tf5nn"
Oct 30 20:01:29.066019 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:29.066016    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p2pv\" (UniqueName: \"kubernetes.io/projected/2225fad5-f918-4bc2-9b8d-c35c84cf570f-kube-api-access-5p2pv\") pod \"maas-api-revoke-sub-1e3a22a6-tf5nn\" (UID: \"2225fad5-f918-4bc2-9b8d-c35c84cf570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a22a6-tf5nn"
Oct 30 20:01:29.066233 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:29.066054    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2225fad5-f918-4bc2-9b8d-c35c84cf570f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e3a22a6-tf5nn\" (UID: \"2225fad5-f918-4bc2-9b8d-c35c84cf570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a22a6-tf5nn"
Oct 30 20:01:29.066814 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:29.066789    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2225fad5-f918-4bc2-9b8d-c35c84cf570f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e3a22a6-tf5nn\" (UID: \"2225fad5-f918-4bc2-9b8d-c35c84cf570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a22a6-tf5nn"
Oct 30 20:01:29.068569 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:29.068552    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2225fad5-f918-4bc2-9b8d-c35c84cf570f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e3a22a6-tf5nn\" (UID: \"2225fad5-f918-4bc2-9b8d-c35c84cf570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a22a6-tf5nn"
Oct 30 20:01:29.127262 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:29.127227    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p2pv\" (UniqueName: \"kubernetes.io/projected/2225fad5-f918-4bc2-9b8d-c35c84cf570f-kube-api-access-5p2pv\") pod \"maas-api-revoke-sub-1e3a22a6-tf5nn\" (UID: \"2225fad5-f918-4bc2-9b8d-c35c84cf570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a22a6-tf5nn"
Oct 30 20:01:29.196613 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:29.196532    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a22a6-tf5nn"
Oct 30 20:01:29.587353 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:29.587329    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e3a22a6-tf5nn"]
Oct 30 20:01:29.589824 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:01:29.589798    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2225fad5_f918_4bc2_9b8d_c35c84cf570f.slice/crio-63cd71adcf892a8f6f79594958318a65f030f4cb8361528146ebec9b6abfe4f3 WatchSource:0}: Error finding container 63cd71adcf892a8f6f79594958318a65f030f4cb8361528146ebec9b6abfe4f3: Status 404 returned error can't find the container with id 63cd71adcf892a8f6f79594958318a65f030f4cb8361528146ebec9b6abfe4f3
Oct 30 20:01:29.772913 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:29.772881    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a22a6-tf5nn" event={"ID":"2225fad5-f918-4bc2-9b8d-c35c84cf570f","Type":"ContainerStarted","Data":"63cd71adcf892a8f6f79594958318a65f030f4cb8361528146ebec9b6abfe4f3"}
Oct 30 20:01:30.780698 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:30.780611    5958 generic.go:356] "Generic (PLEG): container finished" podID="2225fad5-f918-4bc2-9b8d-c35c84cf570f" containerID="001d22446d2f6cdc068b9a38e4760b466c59cd051a865b820f94eca666a4dd85" exitCode=0
Oct 30 20:01:30.781055 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:30.780698    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a22a6-tf5nn" event={"ID":"2225fad5-f918-4bc2-9b8d-c35c84cf570f","Type":"ContainerDied","Data":"001d22446d2f6cdc068b9a38e4760b466c59cd051a865b820f94eca666a4dd85"}
Oct 30 20:01:31.933972 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:31.933936    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a22a6-tf5nn"
Oct 30 20:01:32.103413 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:32.103376    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2225fad5-f918-4bc2-9b8d-c35c84cf570f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2225fad5-f918-4bc2-9b8d-c35c84cf570f-maas-api-openshift-ca\") pod \"2225fad5-f918-4bc2-9b8d-c35c84cf570f\" (UID: \"2225fad5-f918-4bc2-9b8d-c35c84cf570f\") "
Oct 30 20:01:32.103635 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:32.103482    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2225fad5-f918-4bc2-9b8d-c35c84cf570f-kube-api-access-5p2pv\" (UniqueName: \"kubernetes.io/projected/2225fad5-f918-4bc2-9b8d-c35c84cf570f-kube-api-access-5p2pv\") pod \"2225fad5-f918-4bc2-9b8d-c35c84cf570f\" (UID: \"2225fad5-f918-4bc2-9b8d-c35c84cf570f\") "
Oct 30 20:01:32.103635 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:32.103582    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2225fad5-f918-4bc2-9b8d-c35c84cf570f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2225fad5-f918-4bc2-9b8d-c35c84cf570f-maas-api-xks-ca\") pod \"2225fad5-f918-4bc2-9b8d-c35c84cf570f\" (UID: \"2225fad5-f918-4bc2-9b8d-c35c84cf570f\") "
Oct 30 20:01:32.103815 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:32.103785    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2225fad5-f918-4bc2-9b8d-c35c84cf570f-maas-api-openshift-ca" pod "2225fad5-f918-4bc2-9b8d-c35c84cf570f" (UID: "2225fad5-f918-4bc2-9b8d-c35c84cf570f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:32.103945 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:32.103929    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2225fad5-f918-4bc2-9b8d-c35c84cf570f-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:32.105695 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:32.105671    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2225fad5-f918-4bc2-9b8d-c35c84cf570f-maas-api-xks-ca" pod "2225fad5-f918-4bc2-9b8d-c35c84cf570f" (UID: "2225fad5-f918-4bc2-9b8d-c35c84cf570f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:32.105806 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:32.105782    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2225fad5-f918-4bc2-9b8d-c35c84cf570f-kube-api-access-5p2pv" pod "2225fad5-f918-4bc2-9b8d-c35c84cf570f" (UID: "2225fad5-f918-4bc2-9b8d-c35c84cf570f"). InnerVolumeSpecName "kube-api-access-5p2pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:32.205356 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:32.205316    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p2pv\" (UniqueName: \"kubernetes.io/projected/2225fad5-f918-4bc2-9b8d-c35c84cf570f-kube-api-access-5p2pv\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:32.205356 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:32.205352    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2225fad5-f918-4bc2-9b8d-c35c84cf570f-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:32.792857 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:32.792815    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a22a6-tf5nn" event={"ID":"2225fad5-f918-4bc2-9b8d-c35c84cf570f","Type":"ContainerDied","Data":"63cd71adcf892a8f6f79594958318a65f030f4cb8361528146ebec9b6abfe4f3"}
Oct 30 20:01:32.792857 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:32.792838    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a22a6-tf5nn"
Oct 30 20:01:32.793100 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:01:32.792868    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63cd71adcf892a8f6f79594958318a65f030f4cb8361528146ebec9b6abfe4f3"
Oct 30 20:01:36.605412 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:01:36.605356    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:02:06.605741 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:02:06.605698    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:02:15.206785 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.206745    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"]
Oct 30 20:02:15.207294 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.207279    5958 cpu_manager.go:414] "containerMap: removing container" podUID="78f01965-ced2-4a8f-976b-0ac5ed37e6f3" containerName="cleanup"
Oct 30 20:02:15.207294 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.207295    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78f01965-ced2-4a8f-976b-0ac5ed37e6f3" containerName="cleanup"
Oct 30 20:02:15.207372 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.207310    5958 cpu_manager.go:414] "containerMap: removing container" podUID="84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a" containerName="cleanup"
Oct 30 20:02:15.207372 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.207315    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a37f5b-c4f5-4f08-b0ea-420bc56d9e2a" containerName="cleanup"
Oct 30 20:02:15.207372 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.207321    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ec3a0190-8e00-4099-851c-d0c6302899fe" containerName="cleanup"
Oct 30 20:02:15.207372 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.207326    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec3a0190-8e00-4099-851c-d0c6302899fe" containerName="cleanup"
Oct 30 20:02:15.207372 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.207348    5958 cpu_manager.go:414] "containerMap: removing container" podUID="2225fad5-f918-4bc2-9b8d-c35c84cf570f" containerName="revoke-keys"
Oct 30 20:02:15.207372 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.207352    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2225fad5-f918-4bc2-9b8d-c35c84cf570f" containerName="revoke-keys"
Oct 30 20:02:15.207594 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.207461    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec3a0190-8e00-4099-851c-d0c6302899fe" containerName="cleanup"
Oct 30 20:02:15.207594 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.207477    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="78f01965-ced2-4a8f-976b-0ac5ed37e6f3" containerName="cleanup"
Oct 30 20:02:15.207594 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.207489    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="2225fad5-f918-4bc2-9b8d-c35c84cf570f" containerName="revoke-keys"
Oct 30 20:02:15.211236 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.211216    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.268440 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.268404    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-779921-openshift-default-dockercfg-rs78c\""
Oct 30 20:02:15.295791 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.295759    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.295981 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.295822    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.295981 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.295857    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.295981 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.295965    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bfda83c-7138-4974-ae1c-67fa552ac430-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.296113 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.296001    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.296113 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.296021    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.296113 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.296037    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.296207 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.296154    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrm76\" (UniqueName: \"kubernetes.io/projected/8bfda83c-7138-4974-ae1c-67fa552ac430-kube-api-access-rrm76\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.296245 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.296232    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.320905 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.320867    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"]
Oct 30 20:02:15.397088 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.397040    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bfda83c-7138-4974-ae1c-67fa552ac430-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.397258 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.397115    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.397258 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.397143    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.397258 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.397173    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.397258 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.397231    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrm76\" (UniqueName: \"kubernetes.io/projected/8bfda83c-7138-4974-ae1c-67fa552ac430-kube-api-access-rrm76\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.397434 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.397315    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.397530 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.397496    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.397600 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.397583    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.397660 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.397627    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.397809 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.397787    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.397893 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.397850    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.397968 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.397949    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bfda83c-7138-4974-ae1c-67fa552ac430-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.398050 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.398029    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.398248 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.398230    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.400072 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.400048    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.400183 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.400162    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.459420 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.459334    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrm76\" (UniqueName: \"kubernetes.io/projected/8bfda83c-7138-4974-ae1c-67fa552ac430-kube-api-access-rrm76\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.459946 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.459922    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.525869 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.525832    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:15.785493 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.785376    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"]
Oct 30 20:02:15.789905 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.789882    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:15.851766 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.851736    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-bebb42-openshift-default-dockercfg-frbd5\""
Oct 30 20:02:15.857882 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.857850    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"]
Oct 30 20:02:15.904319 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.904282    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:15.904549 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.904334    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:15.904549 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.904412    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:15.904549 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.904445    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:15.904677 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.904574    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:15.904710 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.904677    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:15.904710 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.904699    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:15.904784 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.904736    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:15.904819 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.904800    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5287\" (UniqueName: \"kubernetes.io/projected/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-kube-api-access-w5287\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:15.961395 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.961369    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"]
Oct 30 20:02:15.963623 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:02:15.963593    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bfda83c_7138_4974_ae1c_67fa552ac430.slice/crio-58540b4440d393abac019b6b33c4bfd2b5947000cfcb7b979be82d19d253fdb2 WatchSource:0}: Error finding container 58540b4440d393abac019b6b33c4bfd2b5947000cfcb7b979be82d19d253fdb2: Status 404 returned error can't find the container with id 58540b4440d393abac019b6b33c4bfd2b5947000cfcb7b979be82d19d253fdb2
Oct 30 20:02:15.966287 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.966248    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:02:15.966408 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.966336    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:02:15.966408 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:15.966364    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:02:16.005431 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:16.005401    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:16.005562 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:16.005438    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:16.005562 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:16.005503    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:16.005562 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:16.005530    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5287\" (UniqueName: \"kubernetes.io/projected/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-kube-api-access-w5287\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:16.005719 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:16.005678    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:16.005773 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:16.005745    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:16.005841 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:16.005794    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:16.005841 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:16.005808    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:16.005841 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:16.005832    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:16.006008 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:16.005864    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:16.006008 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:16.005827    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:16.006008 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:16.005989    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:16.006327 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:16.006161    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:16.006408 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:16.006385    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:16.007837 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:16.007816    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:16.008166 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:16.008139    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:16.051639 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:16.051610    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675" event={"ID":"8bfda83c-7138-4974-ae1c-67fa552ac430","Type":"ContainerStarted","Data":"58540b4440d393abac019b6b33c4bfd2b5947000cfcb7b979be82d19d253fdb2"}
Oct 30 20:02:16.067933 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:16.067898    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:16.068063 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:16.067946    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5287\" (UniqueName: \"kubernetes.io/projected/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-kube-api-access-w5287\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:16.103552 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:16.103515    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:16.535343 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:16.535305    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"]
Oct 30 20:02:16.563752 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:16.562521    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:02:16.563752 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:16.562621    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:02:16.563752 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:16.562664    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:02:17.076545 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:17.076440    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc" event={"ID":"76eb3bb9-b9b8-4804-a31b-18165a69ce7f","Type":"ContainerStarted","Data":"b46e13efa9a4b44f447a159a054782d094611c181ae0558bdd5688e70ca25d6b"}
Oct 30 20:02:17.076756 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:17.076579    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc" event={"ID":"76eb3bb9-b9b8-4804-a31b-18165a69ce7f","Type":"ContainerStarted","Data":"74c7dce248e04fb06b7ebedd02a6ffab77d0090c52084b5f99594f8510aa0071"}
Oct 30 20:02:17.078260 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:17.078225    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675" event={"ID":"8bfda83c-7138-4974-ae1c-67fa552ac430","Type":"ContainerStarted","Data":"fbea68500be3cc29b5255741f5efb36efa28803ca2f10d75c30ac94f9bfd9455"}
Oct 30 20:02:17.104314 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:17.104277    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:17.106042 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:17.106011    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 20:02:17.106164 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:17.106077    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc" podUID="76eb3bb9-b9b8-4804-a31b-18165a69ce7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 20:02:17.200311 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:17.200257    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc" podStartSLOduration=2.200240958 podStartE2EDuration="2.200240958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:17.198390238 +0000 UTC m=+1175.930605627" watchObservedRunningTime="2026-10-30 20:02:17.200240958 +0000 UTC m=+1175.932456348"
Oct 30 20:02:17.332412 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:17.332290    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675" podStartSLOduration=2.332269069 podStartE2EDuration="2.332269069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:17.32915748 +0000 UTC m=+1176.061372862" watchObservedRunningTime="2026-10-30 20:02:17.332269069 +0000 UTC m=+1176.064484461"
Oct 30 20:02:17.531476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:17.530310    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:17.531476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:17.530490    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 20:02:17.531476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:17.530538    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675" podUID="8bfda83c-7138-4974-ae1c-67fa552ac430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 20:02:18.104213 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:18.104178    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 20:02:18.104603 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:18.104233    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc" podUID="76eb3bb9-b9b8-4804-a31b-18165a69ce7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 20:02:18.526161 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:18.526070    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 20:02:18.526161 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:18.526128    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675" podUID="8bfda83c-7138-4974-ae1c-67fa552ac430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 20:02:19.104803 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:19.104765    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 20:02:19.105361 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:19.105327    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc" podUID="76eb3bb9-b9b8-4804-a31b-18165a69ce7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 20:02:19.526973 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:19.526868    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 20:02:19.527291 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:19.527261    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675" podUID="8bfda83c-7138-4974-ae1c-67fa552ac430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 20:02:20.104004 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:20.103968    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 20:02:20.104267 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:20.104241    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc" podUID="76eb3bb9-b9b8-4804-a31b-18165a69ce7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 20:02:20.527156 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:20.527061    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 20:02:20.527656 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:20.527623    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675" podUID="8bfda83c-7138-4974-ae1c-67fa552ac430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 20:02:21.104183 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:21.104144    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 20:02:21.104515 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:21.104485    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc" podUID="76eb3bb9-b9b8-4804-a31b-18165a69ce7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 20:02:21.283477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:21.280382    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e21eac26-fpwt9"]
Oct 30 20:02:21.312486 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:21.311793    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21eac26-fpwt9"
Oct 30 20:02:21.394364 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:21.394268    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:02:21.478479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:21.477711    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/000a5b9e-d346-4782-ac64-e31a79f2e9ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e21eac26-fpwt9\" (UID: \"000a5b9e-d346-4782-ac64-e31a79f2e9ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21eac26-fpwt9"
Oct 30 20:02:21.478479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:21.477820    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/000a5b9e-d346-4782-ac64-e31a79f2e9ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e21eac26-fpwt9\" (UID: \"000a5b9e-d346-4782-ac64-e31a79f2e9ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21eac26-fpwt9"
Oct 30 20:02:21.478479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:21.477874    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf74g\" (UniqueName: \"kubernetes.io/projected/000a5b9e-d346-4782-ac64-e31a79f2e9ae-kube-api-access-gf74g\") pod \"maas-api-revoke-sub-e21eac26-fpwt9\" (UID: \"000a5b9e-d346-4782-ac64-e31a79f2e9ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21eac26-fpwt9"
Oct 30 20:02:21.478479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:21.478210    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:02:21.528254 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:21.528217    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 20:02:21.528821 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:21.528788    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675" podUID="8bfda83c-7138-4974-ae1c-67fa552ac430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 20:02:21.588194 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:21.588150    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf74g\" (UniqueName: \"kubernetes.io/projected/000a5b9e-d346-4782-ac64-e31a79f2e9ae-kube-api-access-gf74g\") pod \"maas-api-revoke-sub-e21eac26-fpwt9\" (UID: \"000a5b9e-d346-4782-ac64-e31a79f2e9ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21eac26-fpwt9"
Oct 30 20:02:21.594818 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:21.594787    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/000a5b9e-d346-4782-ac64-e31a79f2e9ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e21eac26-fpwt9\" (UID: \"000a5b9e-d346-4782-ac64-e31a79f2e9ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21eac26-fpwt9"
Oct 30 20:02:21.595113 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:21.595095    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/000a5b9e-d346-4782-ac64-e31a79f2e9ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e21eac26-fpwt9\" (UID: \"000a5b9e-d346-4782-ac64-e31a79f2e9ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21eac26-fpwt9"
Oct 30 20:02:21.599024 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:21.589808    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e21eac26-fpwt9"]
Oct 30 20:02:21.606474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:21.599915    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/000a5b9e-d346-4782-ac64-e31a79f2e9ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e21eac26-fpwt9\" (UID: \"000a5b9e-d346-4782-ac64-e31a79f2e9ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21eac26-fpwt9"
Oct 30 20:02:21.621859 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:21.621831    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/000a5b9e-d346-4782-ac64-e31a79f2e9ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e21eac26-fpwt9\" (UID: \"000a5b9e-d346-4782-ac64-e31a79f2e9ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21eac26-fpwt9"
Oct 30 20:02:21.674575 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:21.663162    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf74g\" (UniqueName: \"kubernetes.io/projected/000a5b9e-d346-4782-ac64-e31a79f2e9ae-kube-api-access-gf74g\") pod \"maas-api-revoke-sub-e21eac26-fpwt9\" (UID: \"000a5b9e-d346-4782-ac64-e31a79f2e9ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21eac26-fpwt9"
Oct 30 20:02:21.953684 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:21.953592    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21eac26-fpwt9"
Oct 30 20:02:22.133493 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:22.131194    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 20:02:22.133493 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:22.131264    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc" podUID="76eb3bb9-b9b8-4804-a31b-18165a69ce7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 20:02:22.527443 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:22.527402    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 20:02:22.527633 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:22.527486    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675" podUID="8bfda83c-7138-4974-ae1c-67fa552ac430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 20:02:22.653525 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:22.653221    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e21eac26-fpwt9"]
Oct 30 20:02:23.105108 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:23.105072    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 20:02:23.105311 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:23.105134    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc" podUID="76eb3bb9-b9b8-4804-a31b-18165a69ce7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 20:02:23.155066 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:23.155024    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21eac26-fpwt9" event={"ID":"000a5b9e-d346-4782-ac64-e31a79f2e9ae","Type":"ContainerStarted","Data":"3323daf91a61017f6a1f6e454aae12fa2e436e78a5d0614818c4b24b31b2e4c2"}
Oct 30 20:02:23.531003 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:23.530977    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:23.531361 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:23.531342    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:23.532040 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:23.532017    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:02:24.164073 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:24.163992    5958 generic.go:356] "Generic (PLEG): container finished" podID="000a5b9e-d346-4782-ac64-e31a79f2e9ae" containerID="551eb85482cd13f3ecfc0c22722f772f31e0dad065c3b2125d657effb2f4b163" exitCode=0
Oct 30 20:02:24.164545 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:24.164075    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21eac26-fpwt9" event={"ID":"000a5b9e-d346-4782-ac64-e31a79f2e9ae","Type":"ContainerDied","Data":"551eb85482cd13f3ecfc0c22722f772f31e0dad065c3b2125d657effb2f4b163"}
Oct 30 20:02:24.701499 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:24.701465    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:24.701821 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:24.701792    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:24.704940 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:24.703293    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:02:25.307356 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:25.307334    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21eac26-fpwt9"
Oct 30 20:02:25.504296 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:25.504210    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/000a5b9e-d346-4782-ac64-e31a79f2e9ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/000a5b9e-d346-4782-ac64-e31a79f2e9ae-maas-api-openshift-ca\") pod \"000a5b9e-d346-4782-ac64-e31a79f2e9ae\" (UID: \"000a5b9e-d346-4782-ac64-e31a79f2e9ae\") "
Oct 30 20:02:25.504483 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:25.504382    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/000a5b9e-d346-4782-ac64-e31a79f2e9ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/000a5b9e-d346-4782-ac64-e31a79f2e9ae-maas-api-xks-ca\") pod \"000a5b9e-d346-4782-ac64-e31a79f2e9ae\" (UID: \"000a5b9e-d346-4782-ac64-e31a79f2e9ae\") "
Oct 30 20:02:25.504483 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:25.504474    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/000a5b9e-d346-4782-ac64-e31a79f2e9ae-kube-api-access-gf74g\" (UniqueName: \"kubernetes.io/projected/000a5b9e-d346-4782-ac64-e31a79f2e9ae-kube-api-access-gf74g\") pod \"000a5b9e-d346-4782-ac64-e31a79f2e9ae\" (UID: \"000a5b9e-d346-4782-ac64-e31a79f2e9ae\") "
Oct 30 20:02:25.504671 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:25.504647    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/000a5b9e-d346-4782-ac64-e31a79f2e9ae-maas-api-openshift-ca" pod "000a5b9e-d346-4782-ac64-e31a79f2e9ae" (UID: "000a5b9e-d346-4782-ac64-e31a79f2e9ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:25.504958 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:25.504921    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/000a5b9e-d346-4782-ac64-e31a79f2e9ae-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:25.506647 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:25.506622    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/000a5b9e-d346-4782-ac64-e31a79f2e9ae-maas-api-xks-ca" pod "000a5b9e-d346-4782-ac64-e31a79f2e9ae" (UID: "000a5b9e-d346-4782-ac64-e31a79f2e9ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:25.506725 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:25.506692    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/000a5b9e-d346-4782-ac64-e31a79f2e9ae-kube-api-access-gf74g" pod "000a5b9e-d346-4782-ac64-e31a79f2e9ae" (UID: "000a5b9e-d346-4782-ac64-e31a79f2e9ae"). InnerVolumeSpecName "kube-api-access-gf74g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:25.608115 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:25.608076    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/000a5b9e-d346-4782-ac64-e31a79f2e9ae-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:25.608325 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:25.608304    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf74g\" (UniqueName: \"kubernetes.io/projected/000a5b9e-d346-4782-ac64-e31a79f2e9ae-kube-api-access-gf74g\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:26.175804 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:26.175767    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21eac26-fpwt9" event={"ID":"000a5b9e-d346-4782-ac64-e31a79f2e9ae","Type":"ContainerDied","Data":"3323daf91a61017f6a1f6e454aae12fa2e436e78a5d0614818c4b24b31b2e4c2"}
Oct 30 20:02:26.175804 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:26.175800    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e21eac26-fpwt9"
Oct 30 20:02:26.176007 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:26.175823    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3323daf91a61017f6a1f6e454aae12fa2e436e78a5d0614818c4b24b31b2e4c2"
Oct 30 20:02:29.543028 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.542994    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"]
Oct 30 20:02:29.543617 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.543598    5958 cpu_manager.go:414] "containerMap: removing container" podUID="000a5b9e-d346-4782-ac64-e31a79f2e9ae" containerName="revoke-keys"
Oct 30 20:02:29.543678 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.543618    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="000a5b9e-d346-4782-ac64-e31a79f2e9ae" containerName="revoke-keys"
Oct 30 20:02:29.543752 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.543740    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="000a5b9e-d346-4782-ac64-e31a79f2e9ae" containerName="revoke-keys"
Oct 30 20:02:29.547108 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.547092    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.599602 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.599566    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-3ccc9f-openshift-default-dockercfg-sn7l4\""
Oct 30 20:02:29.655077 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.655039    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"]
Oct 30 20:02:29.656409 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.656377    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c11b72-fdc6-4d2c-849b-136a5a53259d-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.656628 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.656607    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.656737 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.656722    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.656897 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.656872    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf6z6\" (UniqueName: \"kubernetes.io/projected/48c11b72-fdc6-4d2c-849b-136a5a53259d-kube-api-access-kf6z6\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.657014 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.656938    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.657014 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.656999    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.657104 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.657043    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.657154 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.657105    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.657217 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.657152    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.758266 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.758227    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c11b72-fdc6-4d2c-849b-136a5a53259d-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.758266 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.758265    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.758510 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.758288    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.758510 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.758322    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf6z6\" (UniqueName: \"kubernetes.io/projected/48c11b72-fdc6-4d2c-849b-136a5a53259d-kube-api-access-kf6z6\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.758510 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.758354    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.758510 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.758394    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.758510 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.758433    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.758510 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.758478    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.758814 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.758516    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.758922 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.758900    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.758984 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.758913    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.759022 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.758978    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.759022 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.758991    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.759163 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.759135    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c11b72-fdc6-4d2c-849b-136a5a53259d-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.761084 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.761062    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.761292 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.761274    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.819648 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.819619    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.819798 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.819752    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf6z6\" (UniqueName: \"kubernetes.io/projected/48c11b72-fdc6-4d2c-849b-136a5a53259d-kube-api-access-kf6z6\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:29.861275 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:29.861246    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:30.289400 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:30.289373    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"]
Oct 30 20:02:30.291338 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:02:30.291309    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c11b72_fdc6_4d2c_849b_136a5a53259d.slice/crio-77ed7422a2e5e13dfeb3e1a582d268995f47705a00c2330470e7d6d714485f2c WatchSource:0}: Error finding container 77ed7422a2e5e13dfeb3e1a582d268995f47705a00c2330470e7d6d714485f2c: Status 404 returned error can't find the container with id 77ed7422a2e5e13dfeb3e1a582d268995f47705a00c2330470e7d6d714485f2c
Oct 30 20:02:30.293578 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:30.293544    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:02:30.293654 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:30.293609    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:02:30.293654 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:30.293638    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:02:31.207620 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:31.207575    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf" event={"ID":"48c11b72-fdc6-4d2c-849b-136a5a53259d","Type":"ContainerStarted","Data":"57c38fb01c8ed968f02390fd03d537e541da06a81880e8dac502b98f9ba051cc"}
Oct 30 20:02:31.207982 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:31.207634    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf" event={"ID":"48c11b72-fdc6-4d2c-849b-136a5a53259d","Type":"ContainerStarted","Data":"77ed7422a2e5e13dfeb3e1a582d268995f47705a00c2330470e7d6d714485f2c"}
Oct 30 20:02:31.381308 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:31.381249    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf" podStartSLOduration=2.381232421 podStartE2EDuration="2.381232421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:31.378960476 +0000 UTC m=+1190.111175866" watchObservedRunningTime="2026-10-30 20:02:31.381232421 +0000 UTC m=+1190.113447812"
Oct 30 20:02:31.861836 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:31.861801    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:31.863442 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:31.863408    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 20:02:31.863580 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:31.863486    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf" podUID="48c11b72-fdc6-4d2c-849b-136a5a53259d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 20:02:31.963538 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:31.963154    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3227a6e-mprf8"]
Oct 30 20:02:31.968200 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:31.968176    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3227a6e-mprf8"
Oct 30 20:02:32.017442 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:32.017411    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3227a6e-mprf8"]
Oct 30 20:02:32.020863 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:32.020841    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:02:32.021548 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:32.021528    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:02:32.089848 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:32.089821    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0609b754-971f-4749-87ca-dedea0b03b90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3227a6e-mprf8\" (UID: \"0609b754-971f-4749-87ca-dedea0b03b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3227a6e-mprf8"
Oct 30 20:02:32.090027 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:32.089891    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0609b754-971f-4749-87ca-dedea0b03b90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3227a6e-mprf8\" (UID: \"0609b754-971f-4749-87ca-dedea0b03b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3227a6e-mprf8"
Oct 30 20:02:32.090027 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:32.089952    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7v7j\" (UniqueName: \"kubernetes.io/projected/0609b754-971f-4749-87ca-dedea0b03b90-kube-api-access-b7v7j\") pod \"maas-api-revoke-sub-a3227a6e-mprf8\" (UID: \"0609b754-971f-4749-87ca-dedea0b03b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3227a6e-mprf8"
Oct 30 20:02:32.190946 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:32.190858    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0609b754-971f-4749-87ca-dedea0b03b90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3227a6e-mprf8\" (UID: \"0609b754-971f-4749-87ca-dedea0b03b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3227a6e-mprf8"
Oct 30 20:02:32.190946 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:32.190935    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0609b754-971f-4749-87ca-dedea0b03b90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3227a6e-mprf8\" (UID: \"0609b754-971f-4749-87ca-dedea0b03b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3227a6e-mprf8"
Oct 30 20:02:32.191125 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:32.190953    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7v7j\" (UniqueName: \"kubernetes.io/projected/0609b754-971f-4749-87ca-dedea0b03b90-kube-api-access-b7v7j\") pod \"maas-api-revoke-sub-a3227a6e-mprf8\" (UID: \"0609b754-971f-4749-87ca-dedea0b03b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3227a6e-mprf8"
Oct 30 20:02:32.191643 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:32.191623    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0609b754-971f-4749-87ca-dedea0b03b90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3227a6e-mprf8\" (UID: \"0609b754-971f-4749-87ca-dedea0b03b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3227a6e-mprf8"
Oct 30 20:02:32.193331 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:32.193313    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0609b754-971f-4749-87ca-dedea0b03b90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3227a6e-mprf8\" (UID: \"0609b754-971f-4749-87ca-dedea0b03b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3227a6e-mprf8"
Oct 30 20:02:32.261168 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:32.261136    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7v7j\" (UniqueName: \"kubernetes.io/projected/0609b754-971f-4749-87ca-dedea0b03b90-kube-api-access-b7v7j\") pod \"maas-api-revoke-sub-a3227a6e-mprf8\" (UID: \"0609b754-971f-4749-87ca-dedea0b03b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3227a6e-mprf8"
Oct 30 20:02:32.282424 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:32.282392    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3227a6e-mprf8"
Oct 30 20:02:32.670923 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:32.670897    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3227a6e-mprf8"]
Oct 30 20:02:32.673054 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:02:32.673027    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0609b754_971f_4749_87ca_dedea0b03b90.slice/crio-b58218d16bea9ca63d28e452e4bd79b23c0b85e01da58f7f62026cfbfd3a6711 WatchSource:0}: Error finding container b58218d16bea9ca63d28e452e4bd79b23c0b85e01da58f7f62026cfbfd3a6711: Status 404 returned error can't find the container with id b58218d16bea9ca63d28e452e4bd79b23c0b85e01da58f7f62026cfbfd3a6711
Oct 30 20:02:32.866990 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:32.866948    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 20:02:32.867159 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:32.867030    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf" podUID="48c11b72-fdc6-4d2c-849b-136a5a53259d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 20:02:33.222029 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:33.221993    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3227a6e-mprf8" event={"ID":"0609b754-971f-4749-87ca-dedea0b03b90","Type":"ContainerStarted","Data":"9861fc86be57f0db704d78e824603a7a91bb8bdefa710dea8bee8860ee9df441"}
Oct 30 20:02:33.222207 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:33.222079    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3227a6e-mprf8" event={"ID":"0609b754-971f-4749-87ca-dedea0b03b90","Type":"ContainerStarted","Data":"b58218d16bea9ca63d28e452e4bd79b23c0b85e01da58f7f62026cfbfd3a6711"}
Oct 30 20:02:33.862456 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:33.862408    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 20:02:33.862817 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:33.862475    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf" podUID="48c11b72-fdc6-4d2c-849b-136a5a53259d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 20:02:34.261477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:34.256755    5958 generic.go:356] "Generic (PLEG): container finished" podID="0609b754-971f-4749-87ca-dedea0b03b90" containerID="9861fc86be57f0db704d78e824603a7a91bb8bdefa710dea8bee8860ee9df441" exitCode=0
Oct 30 20:02:34.261477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:34.256961    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3227a6e-mprf8" event={"ID":"0609b754-971f-4749-87ca-dedea0b03b90","Type":"ContainerDied","Data":"9861fc86be57f0db704d78e824603a7a91bb8bdefa710dea8bee8860ee9df441"}
Oct 30 20:02:34.861988 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:34.861951    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 20:02:34.862149 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:34.862018    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf" podUID="48c11b72-fdc6-4d2c-849b-136a5a53259d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 20:02:34.924111 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:34.924077    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"]
Oct 30 20:02:34.928196 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:34.928177    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.020988 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.020951    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc591cde-0da2-429d-8b99-a98f35abadd9-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.021139 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.021086    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc591cde-0da2-429d-8b99-a98f35abadd9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.021209 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.021192    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc591cde-0da2-429d-8b99-a98f35abadd9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.021260 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.021237    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc591cde-0da2-429d-8b99-a98f35abadd9-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.021317 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.021308    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc591cde-0da2-429d-8b99-a98f35abadd9-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.021444 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.021419    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc591cde-0da2-429d-8b99-a98f35abadd9-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.021562 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.021542    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p9gw\" (UniqueName: \"kubernetes.io/projected/cc591cde-0da2-429d-8b99-a98f35abadd9-kube-api-access-7p9gw\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.021619 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.021584    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc591cde-0da2-429d-8b99-a98f35abadd9-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.021687 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.021650    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc591cde-0da2-429d-8b99-a98f35abadd9-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.123354 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.123273    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc591cde-0da2-429d-8b99-a98f35abadd9-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.123354 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.123313    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc591cde-0da2-429d-8b99-a98f35abadd9-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.123566 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.123443    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p9gw\" (UniqueName: \"kubernetes.io/projected/cc591cde-0da2-429d-8b99-a98f35abadd9-kube-api-access-7p9gw\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.123566 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.123519    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc591cde-0da2-429d-8b99-a98f35abadd9-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.123566 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.123545    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc591cde-0da2-429d-8b99-a98f35abadd9-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.123713 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.123590    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc591cde-0da2-429d-8b99-a98f35abadd9-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.123713 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.123652    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc591cde-0da2-429d-8b99-a98f35abadd9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.123713 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.123697    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc591cde-0da2-429d-8b99-a98f35abadd9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.123851 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.123725    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc591cde-0da2-429d-8b99-a98f35abadd9-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.123851 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.123747    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc591cde-0da2-429d-8b99-a98f35abadd9-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.123948 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.123895    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc591cde-0da2-429d-8b99-a98f35abadd9-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.123993 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.123978    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc591cde-0da2-429d-8b99-a98f35abadd9-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.124108 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.124090    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc591cde-0da2-429d-8b99-a98f35abadd9-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.124390 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.124365    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc591cde-0da2-429d-8b99-a98f35abadd9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.125776 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.125754    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc591cde-0da2-429d-8b99-a98f35abadd9-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.126087 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.126071    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc591cde-0da2-429d-8b99-a98f35abadd9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.170830 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.170790    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"]
Oct 30 20:02:35.190417 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.190361    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc591cde-0da2-429d-8b99-a98f35abadd9-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.191322 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.191295    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p9gw\" (UniqueName: \"kubernetes.io/projected/cc591cde-0da2-429d-8b99-a98f35abadd9-kube-api-access-7p9gw\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-5295k\" (UID: \"cc591cde-0da2-429d-8b99-a98f35abadd9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.269286 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.269189    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:35.461503 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.459638    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3227a6e-mprf8"
Oct 30 20:02:35.529613 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.529585    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0609b754-971f-4749-87ca-dedea0b03b90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0609b754-971f-4749-87ca-dedea0b03b90-maas-api-openshift-ca\") pod \"0609b754-971f-4749-87ca-dedea0b03b90\" (UID: \"0609b754-971f-4749-87ca-dedea0b03b90\") "
Oct 30 20:02:35.529773 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.529645    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0609b754-971f-4749-87ca-dedea0b03b90-kube-api-access-b7v7j\" (UniqueName: \"kubernetes.io/projected/0609b754-971f-4749-87ca-dedea0b03b90-kube-api-access-b7v7j\") pod \"0609b754-971f-4749-87ca-dedea0b03b90\" (UID: \"0609b754-971f-4749-87ca-dedea0b03b90\") "
Oct 30 20:02:35.529773 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.529709    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0609b754-971f-4749-87ca-dedea0b03b90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0609b754-971f-4749-87ca-dedea0b03b90-maas-api-xks-ca\") pod \"0609b754-971f-4749-87ca-dedea0b03b90\" (UID: \"0609b754-971f-4749-87ca-dedea0b03b90\") "
Oct 30 20:02:35.530004 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.529972    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0609b754-971f-4749-87ca-dedea0b03b90-maas-api-openshift-ca" pod "0609b754-971f-4749-87ca-dedea0b03b90" (UID: "0609b754-971f-4749-87ca-dedea0b03b90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:35.530270 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.530238    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0609b754-971f-4749-87ca-dedea0b03b90-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:35.531829 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.531804    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0609b754-971f-4749-87ca-dedea0b03b90-kube-api-access-b7v7j" pod "0609b754-971f-4749-87ca-dedea0b03b90" (UID: "0609b754-971f-4749-87ca-dedea0b03b90"). InnerVolumeSpecName "kube-api-access-b7v7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:35.531998 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.531980    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0609b754-971f-4749-87ca-dedea0b03b90-maas-api-xks-ca" pod "0609b754-971f-4749-87ca-dedea0b03b90" (UID: "0609b754-971f-4749-87ca-dedea0b03b90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:35.631381 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.631349    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7v7j\" (UniqueName: \"kubernetes.io/projected/0609b754-971f-4749-87ca-dedea0b03b90-kube-api-access-b7v7j\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:35.631381 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.631378    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0609b754-971f-4749-87ca-dedea0b03b90-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:35.765768 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.765672    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"]
Oct 30 20:02:35.768725 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:02:35.768697    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc591cde_0da2_429d_8b99_a98f35abadd9.slice/crio-e5390f62cc4881f25db3a555d23ff9ec9eeb9f466dcaeb695ecc8c9f00142191 WatchSource:0}: Error finding container e5390f62cc4881f25db3a555d23ff9ec9eeb9f466dcaeb695ecc8c9f00142191: Status 404 returned error can't find the container with id e5390f62cc4881f25db3a555d23ff9ec9eeb9f466dcaeb695ecc8c9f00142191
Oct 30 20:02:35.771885 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.771848    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:02:35.772006 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.771928    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:02:35.772006 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.771972    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:02:35.861919 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.861892    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 20:02:35.862022 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:35.861955    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf" podUID="48c11b72-fdc6-4d2c-849b-136a5a53259d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 20:02:36.274092 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:36.274055    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k" event={"ID":"cc591cde-0da2-429d-8b99-a98f35abadd9","Type":"ContainerStarted","Data":"9a85b2854eed8f66f53aafeee5e04c3b6b4b3d502dcf48640559fbea9631dd34"}
Oct 30 20:02:36.274571 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:36.274113    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k" event={"ID":"cc591cde-0da2-429d-8b99-a98f35abadd9","Type":"ContainerStarted","Data":"e5390f62cc4881f25db3a555d23ff9ec9eeb9f466dcaeb695ecc8c9f00142191"}
Oct 30 20:02:36.275613 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:36.275578    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3227a6e-mprf8" event={"ID":"0609b754-971f-4749-87ca-dedea0b03b90","Type":"ContainerDied","Data":"b58218d16bea9ca63d28e452e4bd79b23c0b85e01da58f7f62026cfbfd3a6711"}
Oct 30 20:02:36.275718 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:36.275639    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b58218d16bea9ca63d28e452e4bd79b23c0b85e01da58f7f62026cfbfd3a6711"
Oct 30 20:02:36.275718 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:36.275599    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3227a6e-mprf8"
Oct 30 20:02:36.401404 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:36.401302    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k" podStartSLOduration=2.401287469 podStartE2EDuration="2.401287469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:36.400339003 +0000 UTC m=+1195.132554395" watchObservedRunningTime="2026-10-30 20:02:36.401287469 +0000 UTC m=+1195.133502858"
Oct 30 20:02:36.611474 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:02:36.611408    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:02:36.878104 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:36.878059    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:36.878998 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:36.878862    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:36.887985 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:36.887953    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:02:37.269742 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:37.269656    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:37.271064 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:37.271035    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5295k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:02:37.271202 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:37.271086    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k" podUID="cc591cde-0da2-429d-8b99-a98f35abadd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 20:02:38.269653 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:38.269606    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5295k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:02:38.270235 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:38.269673    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k" podUID="cc591cde-0da2-429d-8b99-a98f35abadd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 20:02:39.270161 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:39.270125    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5295k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:02:39.270582 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:39.270181    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k" podUID="cc591cde-0da2-429d-8b99-a98f35abadd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 20:02:40.270477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:40.270417    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5295k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:02:40.270897 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:40.270489    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k" podUID="cc591cde-0da2-429d-8b99-a98f35abadd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 20:02:41.270203 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:41.270171    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5295k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:02:41.270384 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:41.270225    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k" podUID="cc591cde-0da2-429d-8b99-a98f35abadd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 20:02:41.593073 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:41.593039    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 20:02:41.595285 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:41.595264    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 20:02:42.270482 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:42.270416    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5295k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:02:42.270788 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:42.270736    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k" podUID="cc591cde-0da2-429d-8b99-a98f35abadd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 20:02:43.272477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:43.272272    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5295k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 20:02:43.272477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:43.272342    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k" podUID="cc591cde-0da2-429d-8b99-a98f35abadd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 20:02:44.761232 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:44.761194    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:44.762689 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:44.762667    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:44.763308 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:44.763288    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k"
Oct 30 20:02:50.038817 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.038780    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"]
Oct 30 20:02:50.039408 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.039389    5958 cpu_manager.go:414] "containerMap: removing container" podUID="0609b754-971f-4749-87ca-dedea0b03b90" containerName="revoke-keys"
Oct 30 20:02:50.039494 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.039412    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0609b754-971f-4749-87ca-dedea0b03b90" containerName="revoke-keys"
Oct 30 20:02:50.039693 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.039680    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="0609b754-971f-4749-87ca-dedea0b03b90" containerName="revoke-keys"
Oct 30 20:02:50.044727 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.044709    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.078417 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.078383    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.078561 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.078482    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.078640 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.078620    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.078688 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.078658    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.078767 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.078747    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfdt9\" (UniqueName: \"kubernetes.io/projected/3e3b7c1d-c991-4375-8caa-df21545e1fd8-kube-api-access-kfdt9\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.078830 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.078788    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.078887 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.078875    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.078935 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.078895    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.079021 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.079006    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.151603 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.151572    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"]
Oct 30 20:02:50.180511 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.180480    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.180687 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.180540    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.180687 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.180575    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfdt9\" (UniqueName: \"kubernetes.io/projected/3e3b7c1d-c991-4375-8caa-df21545e1fd8-kube-api-access-kfdt9\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.180687 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.180599    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.180687 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.180646    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.180687 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.180668    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.180937 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.180738    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.180937 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.180834    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.180937 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.180880    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.181079 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.180943    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.181133 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.181103    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.181215 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.181193    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.181335 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.181317    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.181716 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.181690    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.182948 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.182928    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.183174 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.183158    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.268685 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.268653    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.268685 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.268671    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfdt9\" (UniqueName: \"kubernetes.io/projected/3e3b7c1d-c991-4375-8caa-df21545e1fd8-kube-api-access-kfdt9\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xf8pl\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.357799 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.357773    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:50.787652 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.787627    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"]
Oct 30 20:02:50.793756 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.793719    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:02:50.793860 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.793790    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:02:50.793860 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:50.793819    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:02:51.398739 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:51.398707    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl" event={"ID":"3e3b7c1d-c991-4375-8caa-df21545e1fd8","Type":"ContainerStarted","Data":"8f096bee4e36f6a8e26b25492fea44706b34ea384c3d6c44fa2783fed74f1a8c"}
Oct 30 20:02:51.399113 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:51.398752    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl" event={"ID":"3e3b7c1d-c991-4375-8caa-df21545e1fd8","Type":"ContainerStarted","Data":"74e492458dc73520a63c4bc834e5655d8ce0954c67fb75b40fe6536d989864e9"}
Oct 30 20:02:51.522440 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:51.522377    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl" podStartSLOduration=2.522358489 podStartE2EDuration="2.522358489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:51.521797048 +0000 UTC m=+1210.254012438" watchObservedRunningTime="2026-10-30 20:02:51.522358489 +0000 UTC m=+1210.254573878"
Oct 30 20:02:52.360744 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:52.360695    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:52.362279 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:52.362254    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 20:02:52.362354 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:52.362305    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl" podUID="3e3b7c1d-c991-4375-8caa-df21545e1fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 20:02:53.360813 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:53.360783    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 20:02:53.361208 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:53.360846    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl" podUID="3e3b7c1d-c991-4375-8caa-df21545e1fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 20:02:54.360857 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:54.360813    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 20:02:54.361334 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:54.360897    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl" podUID="3e3b7c1d-c991-4375-8caa-df21545e1fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 20:02:55.360481 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:55.358813    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 20:02:55.360481 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:55.358882    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl" podUID="3e3b7c1d-c991-4375-8caa-df21545e1fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 20:02:56.360954 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:56.360909    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 20:02:56.361365 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:56.360978    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl" podUID="3e3b7c1d-c991-4375-8caa-df21545e1fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 20:02:57.359133 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:57.359094    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 20:02:57.359320 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:57.359156    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl" podUID="3e3b7c1d-c991-4375-8caa-df21545e1fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 20:02:58.925773 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:58.925735    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:58.926275 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:58.926214    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:02:58.927028 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:02:58.927003    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:03:06.615707 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:03:06.615669    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:03:14.432329 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.432296    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"]
Oct 30 20:03:14.438835 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.438768    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.497153 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.497118    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.497345 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.497218    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.497345 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.497265    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.497587 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.497566    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.497656 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.497613    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.497700 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.497676    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551cc517-12e5-49b1-beca-2ca8348fd2b8-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.497863 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.497845    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.497985 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.497964    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65xq4\" (UniqueName: \"kubernetes.io/projected/551cc517-12e5-49b1-beca-2ca8348fd2b8-kube-api-access-65xq4\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.498098 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.498077    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.544794 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.544760    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"]
Oct 30 20:03:14.599406 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.599369    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65xq4\" (UniqueName: \"kubernetes.io/projected/551cc517-12e5-49b1-beca-2ca8348fd2b8-kube-api-access-65xq4\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.599682 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.599436    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.599682 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.599508    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.599682 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.599542    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.599682 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.599682    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.599902 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.599721    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.599902 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.599748    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.599902 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.599765    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551cc517-12e5-49b1-beca-2ca8348fd2b8-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.599902 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.599831    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.600073 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.599945    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.600073 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.599992    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.600073 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.600045    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.600173 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.600094    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.600419 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.600398    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551cc517-12e5-49b1-beca-2ca8348fd2b8-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.602167 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.602148    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.602367 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.602350    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.661393 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.661348    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65xq4\" (UniqueName: \"kubernetes.io/projected/551cc517-12e5-49b1-beca-2ca8348fd2b8-kube-api-access-65xq4\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.661990 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.661971    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:14.760493 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:14.760391    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:15.196524 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:15.196497    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"]
Oct 30 20:03:15.198040 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:03:15.198011    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551cc517_12e5_49b1_beca_2ca8348fd2b8.slice/crio-6521a6eeb75f86faf9e556be8f4a94624c54de2d5e696a5218b1e4328b57d039 WatchSource:0}: Error finding container 6521a6eeb75f86faf9e556be8f4a94624c54de2d5e696a5218b1e4328b57d039: Status 404 returned error can't find the container with id 6521a6eeb75f86faf9e556be8f4a94624c54de2d5e696a5218b1e4328b57d039
Oct 30 20:03:15.200598 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:15.200565    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:03:15.200673 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:15.200634    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:03:15.200673 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:15.200659    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:03:15.558540 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:15.558431    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv" event={"ID":"551cc517-12e5-49b1-beca-2ca8348fd2b8","Type":"ContainerStarted","Data":"5edc3281806b6a7be21e6ca69a21324c0ae73e6817a3b0e21bfd842e2964b45f"}
Oct 30 20:03:15.558540 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:15.558506    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv" event={"ID":"551cc517-12e5-49b1-beca-2ca8348fd2b8","Type":"ContainerStarted","Data":"6521a6eeb75f86faf9e556be8f4a94624c54de2d5e696a5218b1e4328b57d039"}
Oct 30 20:03:15.689615 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:15.689558    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv" podStartSLOduration=1.6895428080000001 podStartE2EDuration="1.689542808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:15.686232632 +0000 UTC m=+1234.418448025" watchObservedRunningTime="2026-10-30 20:03:15.689542808 +0000 UTC m=+1234.421758198"
Oct 30 20:03:15.761951 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:15.761715    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:16.763246 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:16.763205    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 20:03:16.763246 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:16.763276    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv" podUID="551cc517-12e5-49b1-beca-2ca8348fd2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 20:03:17.761860 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:17.761809    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 20:03:17.762069 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:17.761895    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv" podUID="551cc517-12e5-49b1-beca-2ca8348fd2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 20:03:18.761980 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:18.761943    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 20:03:18.762488 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:18.762136    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv" podUID="551cc517-12e5-49b1-beca-2ca8348fd2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 20:03:19.761933 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:19.761884    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 20:03:19.762140 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:19.761963    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv" podUID="551cc517-12e5-49b1-beca-2ca8348fd2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 20:03:20.761628 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:20.761594    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 20:03:20.761807 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:20.761652    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv" podUID="551cc517-12e5-49b1-beca-2ca8348fd2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 20:03:21.815224 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:21.815191    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:21.815735 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:21.815596    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:21.817379 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:21.817356    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:03:36.606731 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:03:36.606694    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:03:38.840324 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:38.840291    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"]
Oct 30 20:03:38.844360 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:38.844340    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:03:38.910559 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:38.910523    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-tls-certs\") pod \"distinct-w6-f75272-kserve-7fb9554bc4-bvnk6\" (UID: \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:03:38.910877 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:38.910850    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-tmp-dir\") pod \"distinct-w6-f75272-kserve-7fb9554bc4-bvnk6\" (UID: \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:03:38.910998 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:38.910911    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-model-cache\") pod \"distinct-w6-f75272-kserve-7fb9554bc4-bvnk6\" (UID: \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:03:38.911063 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:38.910988    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-home\") pod \"distinct-w6-f75272-kserve-7fb9554bc4-bvnk6\" (UID: \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:03:38.911063 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:38.911047    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-dshm\") pod \"distinct-w6-f75272-kserve-7fb9554bc4-bvnk6\" (UID: \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:03:38.911162 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:38.911080    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5tb9\" (UniqueName: \"kubernetes.io/projected/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-kube-api-access-p5tb9\") pod \"distinct-w6-f75272-kserve-7fb9554bc4-bvnk6\" (UID: \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:03:38.947354 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:38.947316    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f75272\"/\"distinct-w6-f75272-kserve-self-signed-certs\""
Oct 30 20:03:38.949977 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:38.949954    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"]
Oct 30 20:03:39.013443 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:39.013402    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-tmp-dir\") pod \"distinct-w6-f75272-kserve-7fb9554bc4-bvnk6\" (UID: \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:03:39.013770 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:39.013739    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-model-cache\") pod \"distinct-w6-f75272-kserve-7fb9554bc4-bvnk6\" (UID: \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:03:39.013894 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:39.013801    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-home\") pod \"distinct-w6-f75272-kserve-7fb9554bc4-bvnk6\" (UID: \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:03:39.013894 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:39.013873    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-dshm\") pod \"distinct-w6-f75272-kserve-7fb9554bc4-bvnk6\" (UID: \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:03:39.013991 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:39.013923    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-tmp-dir\") pod \"distinct-w6-f75272-kserve-7fb9554bc4-bvnk6\" (UID: \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:03:39.013991 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:39.013937    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5tb9\" (UniqueName: \"kubernetes.io/projected/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-kube-api-access-p5tb9\") pod \"distinct-w6-f75272-kserve-7fb9554bc4-bvnk6\" (UID: \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:03:39.014204 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:39.014182    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-model-cache\") pod \"distinct-w6-f75272-kserve-7fb9554bc4-bvnk6\" (UID: \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:03:39.015173 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:39.015022    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-tls-certs\") pod \"distinct-w6-f75272-kserve-7fb9554bc4-bvnk6\" (UID: \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:03:39.017015 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:39.016991    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-home\") pod \"distinct-w6-f75272-kserve-7fb9554bc4-bvnk6\" (UID: \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:03:39.020153 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:39.020124    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-tls-certs\") pod \"distinct-w6-f75272-kserve-7fb9554bc4-bvnk6\" (UID: \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:03:39.023647 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:39.023625    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-dshm\") pod \"distinct-w6-f75272-kserve-7fb9554bc4-bvnk6\" (UID: \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:03:39.087480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:39.085392    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5tb9\" (UniqueName: \"kubernetes.io/projected/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-kube-api-access-p5tb9\") pod \"distinct-w6-f75272-kserve-7fb9554bc4-bvnk6\" (UID: \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\") " pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:03:39.157969 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:39.157877    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:03:39.576119 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:39.576096    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"]
Oct 30 20:03:39.578427 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:03:39.578399    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb0f89dc_ae94_44d8_ac5c_f7aa9cfae480.slice/crio-c49c930a42bef8836f28f701f56c2769b0a4bfc876928dc5206aa86669c0c21a WatchSource:0}: Error finding container c49c930a42bef8836f28f701f56c2769b0a4bfc876928dc5206aa86669c0c21a: Status 404 returned error can't find the container with id c49c930a42bef8836f28f701f56c2769b0a4bfc876928dc5206aa86669c0c21a
Oct 30 20:03:39.712665 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:39.712625    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" event={"ID":"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480","Type":"ContainerStarted","Data":"4d4201fe8e4f622e949bd0199f54266951a843b213469b78b869852ddf1112c1"}
Oct 30 20:03:39.712830 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:39.712680    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" event={"ID":"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480","Type":"ContainerStarted","Data":"c49c930a42bef8836f28f701f56c2769b0a4bfc876928dc5206aa86669c0c21a"}
Oct 30 20:03:39.831831 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:39.831726    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podStartSLOduration=1.831708557 podStartE2EDuration="1.831708557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:39.830012993 +0000 UTC m=+1258.562228385" watchObservedRunningTime="2026-10-30 20:03:39.831708557 +0000 UTC m=+1258.563923953"
Oct 30 20:03:40.158300 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:40.158217    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:03:49.158886 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:49.158847    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:03:49.171661 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:49.171635    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:03:49.783096 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:03:49.783028    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:04:06.604753 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:04:06.604714    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:04:14.281156 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:14.281124    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d96a36be-tz9jd"]
Oct 30 20:04:14.285277 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:14.285258    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96a36be-tz9jd"
Oct 30 20:04:14.344296 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:14.344268    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d96a36be-tz9jd"]
Oct 30 20:04:14.389716 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:14.389686    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ba655b7-a698-4157-91c9-5849e1b6bbd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d96a36be-tz9jd\" (UID: \"1ba655b7-a698-4157-91c9-5849e1b6bbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96a36be-tz9jd"
Oct 30 20:04:14.389890 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:14.389845    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqrxg\" (UniqueName: \"kubernetes.io/projected/1ba655b7-a698-4157-91c9-5849e1b6bbd6-kube-api-access-lqrxg\") pod \"maas-api-revoke-sub-d96a36be-tz9jd\" (UID: \"1ba655b7-a698-4157-91c9-5849e1b6bbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96a36be-tz9jd"
Oct 30 20:04:14.389957 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:14.389918    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ba655b7-a698-4157-91c9-5849e1b6bbd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d96a36be-tz9jd\" (UID: \"1ba655b7-a698-4157-91c9-5849e1b6bbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96a36be-tz9jd"
Oct 30 20:04:14.390946 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:14.390925    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:04:14.391007 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:14.390942    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:04:14.490875 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:14.490847    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ba655b7-a698-4157-91c9-5849e1b6bbd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d96a36be-tz9jd\" (UID: \"1ba655b7-a698-4157-91c9-5849e1b6bbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96a36be-tz9jd"
Oct 30 20:04:14.491020 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:14.490951    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqrxg\" (UniqueName: \"kubernetes.io/projected/1ba655b7-a698-4157-91c9-5849e1b6bbd6-kube-api-access-lqrxg\") pod \"maas-api-revoke-sub-d96a36be-tz9jd\" (UID: \"1ba655b7-a698-4157-91c9-5849e1b6bbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96a36be-tz9jd"
Oct 30 20:04:14.491020 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:14.491001    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ba655b7-a698-4157-91c9-5849e1b6bbd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d96a36be-tz9jd\" (UID: \"1ba655b7-a698-4157-91c9-5849e1b6bbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96a36be-tz9jd"
Oct 30 20:04:14.491580 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:14.491558    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ba655b7-a698-4157-91c9-5849e1b6bbd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d96a36be-tz9jd\" (UID: \"1ba655b7-a698-4157-91c9-5849e1b6bbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96a36be-tz9jd"
Oct 30 20:04:14.493339 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:14.493316    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ba655b7-a698-4157-91c9-5849e1b6bbd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d96a36be-tz9jd\" (UID: \"1ba655b7-a698-4157-91c9-5849e1b6bbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96a36be-tz9jd"
Oct 30 20:04:14.551826 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:14.551763    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqrxg\" (UniqueName: \"kubernetes.io/projected/1ba655b7-a698-4157-91c9-5849e1b6bbd6-kube-api-access-lqrxg\") pod \"maas-api-revoke-sub-d96a36be-tz9jd\" (UID: \"1ba655b7-a698-4157-91c9-5849e1b6bbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96a36be-tz9jd"
Oct 30 20:04:14.597960 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:14.597941    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96a36be-tz9jd"
Oct 30 20:04:14.973107 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:14.973075    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d96a36be-tz9jd"]
Oct 30 20:04:14.974289 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:04:14.974257    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba655b7_a698_4157_91c9_5849e1b6bbd6.slice/crio-21f72fe5a8cbc122fae47c0133783c61a7a4b35ced67b09a47161f8927e4eed8 WatchSource:0}: Error finding container 21f72fe5a8cbc122fae47c0133783c61a7a4b35ced67b09a47161f8927e4eed8: Status 404 returned error can't find the container with id 21f72fe5a8cbc122fae47c0133783c61a7a4b35ced67b09a47161f8927e4eed8
Oct 30 20:04:15.931702 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:15.931674    5958 generic.go:356] "Generic (PLEG): container finished" podID="1ba655b7-a698-4157-91c9-5849e1b6bbd6" containerID="d0b8a5e1570b2bf9169cd752a29b31c41a453f37d862360f794d7201b9868132" exitCode=0
Oct 30 20:04:15.932034 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:15.931739    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96a36be-tz9jd" event={"ID":"1ba655b7-a698-4157-91c9-5849e1b6bbd6","Type":"ContainerDied","Data":"d0b8a5e1570b2bf9169cd752a29b31c41a453f37d862360f794d7201b9868132"}
Oct 30 20:04:15.932034 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:15.931774    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96a36be-tz9jd" event={"ID":"1ba655b7-a698-4157-91c9-5849e1b6bbd6","Type":"ContainerStarted","Data":"21f72fe5a8cbc122fae47c0133783c61a7a4b35ced67b09a47161f8927e4eed8"}
Oct 30 20:04:17.074899 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:17.074875    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96a36be-tz9jd"
Oct 30 20:04:17.116335 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:17.116312    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ba655b7-a698-4157-91c9-5849e1b6bbd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ba655b7-a698-4157-91c9-5849e1b6bbd6-maas-api-xks-ca\") pod \"1ba655b7-a698-4157-91c9-5849e1b6bbd6\" (UID: \"1ba655b7-a698-4157-91c9-5849e1b6bbd6\") "
Oct 30 20:04:17.116478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:17.116366    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba655b7-a698-4157-91c9-5849e1b6bbd6-kube-api-access-lqrxg\" (UniqueName: \"kubernetes.io/projected/1ba655b7-a698-4157-91c9-5849e1b6bbd6-kube-api-access-lqrxg\") pod \"1ba655b7-a698-4157-91c9-5849e1b6bbd6\" (UID: \"1ba655b7-a698-4157-91c9-5849e1b6bbd6\") "
Oct 30 20:04:17.116478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:17.116415    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ba655b7-a698-4157-91c9-5849e1b6bbd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ba655b7-a698-4157-91c9-5849e1b6bbd6-maas-api-openshift-ca\") pod \"1ba655b7-a698-4157-91c9-5849e1b6bbd6\" (UID: \"1ba655b7-a698-4157-91c9-5849e1b6bbd6\") "
Oct 30 20:04:17.116905 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:17.116876    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba655b7-a698-4157-91c9-5849e1b6bbd6-maas-api-openshift-ca" pod "1ba655b7-a698-4157-91c9-5849e1b6bbd6" (UID: "1ba655b7-a698-4157-91c9-5849e1b6bbd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:17.118649 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:17.118623    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba655b7-a698-4157-91c9-5849e1b6bbd6-kube-api-access-lqrxg" pod "1ba655b7-a698-4157-91c9-5849e1b6bbd6" (UID: "1ba655b7-a698-4157-91c9-5849e1b6bbd6"). InnerVolumeSpecName "kube-api-access-lqrxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:17.118921 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:17.118897    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ba655b7-a698-4157-91c9-5849e1b6bbd6-maas-api-xks-ca" pod "1ba655b7-a698-4157-91c9-5849e1b6bbd6" (UID: "1ba655b7-a698-4157-91c9-5849e1b6bbd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:04:17.217882 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:17.217810    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ba655b7-a698-4157-91c9-5849e1b6bbd6-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:17.217882 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:17.217837    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqrxg\" (UniqueName: \"kubernetes.io/projected/1ba655b7-a698-4157-91c9-5849e1b6bbd6-kube-api-access-lqrxg\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:17.217882 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:17.217848    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ba655b7-a698-4157-91c9-5849e1b6bbd6-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:17.943921 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:17.943892    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96a36be-tz9jd"
Oct 30 20:04:17.943921 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:17.943906    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d96a36be-tz9jd" event={"ID":"1ba655b7-a698-4157-91c9-5849e1b6bbd6","Type":"ContainerDied","Data":"21f72fe5a8cbc122fae47c0133783c61a7a4b35ced67b09a47161f8927e4eed8"}
Oct 30 20:04:17.944116 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:17.943948    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21f72fe5a8cbc122fae47c0133783c61a7a4b35ced67b09a47161f8927e4eed8"
Oct 30 20:04:18.051484 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.051439    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"]
Oct 30 20:04:18.052085 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.052066    5958 cpu_manager.go:414] "containerMap: removing container" podUID="1ba655b7-a698-4157-91c9-5849e1b6bbd6" containerName="revoke-keys"
Oct 30 20:04:18.052174 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.052087    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ba655b7-a698-4157-91c9-5849e1b6bbd6" containerName="revoke-keys"
Oct 30 20:04:18.052265 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.052251    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ba655b7-a698-4157-91c9-5849e1b6bbd6" containerName="revoke-keys"
Oct 30 20:04:18.055688 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.055668    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:04:18.109715 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.109688    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-779921\"/\"distinct-2-w0-779921-kserve-self-signed-certs\""
Oct 30 20:04:18.112939 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.112914    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"]
Oct 30 20:04:18.130441 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.130412    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-home\") pod \"distinct-2-w0-779921-kserve-58f78dfb47-vmpsl\" (UID: \"3fff08e6-97c4-4572-9aad-95ae489fbffb\") " pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:04:18.130660 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.130562    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-dshm\") pod \"distinct-2-w0-779921-kserve-58f78dfb47-vmpsl\" (UID: \"3fff08e6-97c4-4572-9aad-95ae489fbffb\") " pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:04:18.130778 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.130739    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-tmp-dir\") pod \"distinct-2-w0-779921-kserve-58f78dfb47-vmpsl\" (UID: \"3fff08e6-97c4-4572-9aad-95ae489fbffb\") " pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:04:18.130843 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.130773    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqktl\" (UniqueName: \"kubernetes.io/projected/3fff08e6-97c4-4572-9aad-95ae489fbffb-kube-api-access-rqktl\") pod \"distinct-2-w0-779921-kserve-58f78dfb47-vmpsl\" (UID: \"3fff08e6-97c4-4572-9aad-95ae489fbffb\") " pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:04:18.130843 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.130824    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-model-cache\") pod \"distinct-2-w0-779921-kserve-58f78dfb47-vmpsl\" (UID: \"3fff08e6-97c4-4572-9aad-95ae489fbffb\") " pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:04:18.130955 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.130932    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fff08e6-97c4-4572-9aad-95ae489fbffb-tls-certs\") pod \"distinct-2-w0-779921-kserve-58f78dfb47-vmpsl\" (UID: \"3fff08e6-97c4-4572-9aad-95ae489fbffb\") " pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:04:18.232518 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.232409    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-tmp-dir\") pod \"distinct-2-w0-779921-kserve-58f78dfb47-vmpsl\" (UID: \"3fff08e6-97c4-4572-9aad-95ae489fbffb\") " pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:04:18.232785 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.232757    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqktl\" (UniqueName: \"kubernetes.io/projected/3fff08e6-97c4-4572-9aad-95ae489fbffb-kube-api-access-rqktl\") pod \"distinct-2-w0-779921-kserve-58f78dfb47-vmpsl\" (UID: \"3fff08e6-97c4-4572-9aad-95ae489fbffb\") " pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:04:18.232976 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.232960    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-model-cache\") pod \"distinct-2-w0-779921-kserve-58f78dfb47-vmpsl\" (UID: \"3fff08e6-97c4-4572-9aad-95ae489fbffb\") " pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:04:18.233079 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.233066    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fff08e6-97c4-4572-9aad-95ae489fbffb-tls-certs\") pod \"distinct-2-w0-779921-kserve-58f78dfb47-vmpsl\" (UID: \"3fff08e6-97c4-4572-9aad-95ae489fbffb\") " pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:04:18.233406 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.233388    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-home\") pod \"distinct-2-w0-779921-kserve-58f78dfb47-vmpsl\" (UID: \"3fff08e6-97c4-4572-9aad-95ae489fbffb\") " pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:04:18.233618 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.233601    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-dshm\") pod \"distinct-2-w0-779921-kserve-58f78dfb47-vmpsl\" (UID: \"3fff08e6-97c4-4572-9aad-95ae489fbffb\") " pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:04:18.233866 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.233816    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-tmp-dir\") pod \"distinct-2-w0-779921-kserve-58f78dfb47-vmpsl\" (UID: \"3fff08e6-97c4-4572-9aad-95ae489fbffb\") " pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:04:18.234137 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.233944    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-model-cache\") pod \"distinct-2-w0-779921-kserve-58f78dfb47-vmpsl\" (UID: \"3fff08e6-97c4-4572-9aad-95ae489fbffb\") " pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:04:18.234565 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.234520    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-home\") pod \"distinct-2-w0-779921-kserve-58f78dfb47-vmpsl\" (UID: \"3fff08e6-97c4-4572-9aad-95ae489fbffb\") " pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:04:18.238881 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.237841    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fff08e6-97c4-4572-9aad-95ae489fbffb-tls-certs\") pod \"distinct-2-w0-779921-kserve-58f78dfb47-vmpsl\" (UID: \"3fff08e6-97c4-4572-9aad-95ae489fbffb\") " pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:04:18.239495 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.239439    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-dshm\") pod \"distinct-2-w0-779921-kserve-58f78dfb47-vmpsl\" (UID: \"3fff08e6-97c4-4572-9aad-95ae489fbffb\") " pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:04:18.295223 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.295195    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqktl\" (UniqueName: \"kubernetes.io/projected/3fff08e6-97c4-4572-9aad-95ae489fbffb-kube-api-access-rqktl\") pod \"distinct-2-w0-779921-kserve-58f78dfb47-vmpsl\" (UID: \"3fff08e6-97c4-4572-9aad-95ae489fbffb\") " pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:04:18.367510 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.367482    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:04:18.777109 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.777081    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"]
Oct 30 20:04:18.778747 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:04:18.778724    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fff08e6_97c4_4572_9aad_95ae489fbffb.slice/crio-efa39610e8fa9638240014b08abc7b80ed09e512c3abd139f9a130c6daef116c WatchSource:0}: Error finding container efa39610e8fa9638240014b08abc7b80ed09e512c3abd139f9a130c6daef116c: Status 404 returned error can't find the container with id efa39610e8fa9638240014b08abc7b80ed09e512c3abd139f9a130c6daef116c
Oct 30 20:04:18.952024 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.951977    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" event={"ID":"3fff08e6-97c4-4572-9aad-95ae489fbffb","Type":"ContainerStarted","Data":"8518e28d69355a3546b16bbd82191ffbc89666665a0ee210293d7f14123630ba"}
Oct 30 20:04:18.952150 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:18.952031    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" event={"ID":"3fff08e6-97c4-4572-9aad-95ae489fbffb","Type":"ContainerStarted","Data":"efa39610e8fa9638240014b08abc7b80ed09e512c3abd139f9a130c6daef116c"}
Oct 30 20:04:19.075479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:19.075398    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" podStartSLOduration=1.075385877 podStartE2EDuration="1.075385877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:19.074319877 +0000 UTC m=+1297.806535296" watchObservedRunningTime="2026-10-30 20:04:19.075385877 +0000 UTC m=+1297.807601266"
Oct 30 20:04:19.368522 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:19.368437    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:04:27.809654 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:27.809619    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52d5c320-688gn"]
Oct 30 20:04:27.813656 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:27.813636    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d5c320-688gn"
Oct 30 20:04:27.918278 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:27.918249    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:04:27.918278 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:27.918249    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:04:27.921060 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:27.921037    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52d5c320-688gn"]
Oct 30 20:04:27.959203 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:27.959174    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b73e696-124a-4cab-bb04-442487660026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52d5c320-688gn\" (UID: \"0b73e696-124a-4cab-bb04-442487660026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d5c320-688gn"
Oct 30 20:04:27.959322 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:27.959307    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b73e696-124a-4cab-bb04-442487660026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52d5c320-688gn\" (UID: \"0b73e696-124a-4cab-bb04-442487660026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d5c320-688gn"
Oct 30 20:04:27.959394 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:27.959372    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqmvr\" (UniqueName: \"kubernetes.io/projected/0b73e696-124a-4cab-bb04-442487660026-kube-api-access-lqmvr\") pod \"maas-api-revoke-sub-52d5c320-688gn\" (UID: \"0b73e696-124a-4cab-bb04-442487660026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d5c320-688gn"
Oct 30 20:04:28.060769 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:28.060674    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b73e696-124a-4cab-bb04-442487660026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52d5c320-688gn\" (UID: \"0b73e696-124a-4cab-bb04-442487660026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d5c320-688gn"
Oct 30 20:04:28.060769 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:28.060730    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqmvr\" (UniqueName: \"kubernetes.io/projected/0b73e696-124a-4cab-bb04-442487660026-kube-api-access-lqmvr\") pod \"maas-api-revoke-sub-52d5c320-688gn\" (UID: \"0b73e696-124a-4cab-bb04-442487660026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d5c320-688gn"
Oct 30 20:04:28.060998 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:28.060908    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b73e696-124a-4cab-bb04-442487660026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52d5c320-688gn\" (UID: \"0b73e696-124a-4cab-bb04-442487660026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d5c320-688gn"
Oct 30 20:04:28.061462 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:28.061426    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b73e696-124a-4cab-bb04-442487660026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52d5c320-688gn\" (UID: \"0b73e696-124a-4cab-bb04-442487660026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d5c320-688gn"
Oct 30 20:04:28.063407 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:28.063384    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b73e696-124a-4cab-bb04-442487660026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52d5c320-688gn\" (UID: \"0b73e696-124a-4cab-bb04-442487660026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d5c320-688gn"
Oct 30 20:04:28.123876 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:28.123856    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqmvr\" (UniqueName: \"kubernetes.io/projected/0b73e696-124a-4cab-bb04-442487660026-kube-api-access-lqmvr\") pod \"maas-api-revoke-sub-52d5c320-688gn\" (UID: \"0b73e696-124a-4cab-bb04-442487660026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d5c320-688gn"
Oct 30 20:04:28.127570 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:28.127547    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d5c320-688gn"
Oct 30 20:04:28.367625 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:28.367596    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:04:28.383582 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:28.383557    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:04:28.503156 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:28.503129    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52d5c320-688gn"]
Oct 30 20:04:28.504715 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:04:28.504688    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b73e696_124a_4cab_bb04_442487660026.slice/crio-3041fe007fffd7fcbd6031189c4292916bb8c0f56eee1024569c347540784bbb WatchSource:0}: Error finding container 3041fe007fffd7fcbd6031189c4292916bb8c0f56eee1024569c347540784bbb: Status 404 returned error can't find the container with id 3041fe007fffd7fcbd6031189c4292916bb8c0f56eee1024569c347540784bbb
Oct 30 20:04:29.015100 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:29.015024    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d5c320-688gn" event={"ID":"0b73e696-124a-4cab-bb04-442487660026","Type":"ContainerStarted","Data":"395648b038f0b43aeee8938954d7a04fd7241a9e1e9a87d4e89f5ab372b85dad"}
Oct 30 20:04:29.015100 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:29.015086    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d5c320-688gn" event={"ID":"0b73e696-124a-4cab-bb04-442487660026","Type":"ContainerStarted","Data":"3041fe007fffd7fcbd6031189c4292916bb8c0f56eee1024569c347540784bbb"}
Oct 30 20:04:29.025750 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:29.025730    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:04:29.255596 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:29.255543    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d5c320-688gn" podStartSLOduration=2.255528543 podStartE2EDuration="2.255528543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:29.135546928 +0000 UTC m=+1307.867762317" watchObservedRunningTime="2026-10-30 20:04:29.255528543 +0000 UTC m=+1307.987743933"
Oct 30 20:04:30.023442 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:30.023397    5958 generic.go:356] "Generic (PLEG): container finished" podID="0b73e696-124a-4cab-bb04-442487660026" containerID="395648b038f0b43aeee8938954d7a04fd7241a9e1e9a87d4e89f5ab372b85dad" exitCode=0
Oct 30 20:04:30.023913 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:30.023485    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d5c320-688gn" event={"ID":"0b73e696-124a-4cab-bb04-442487660026","Type":"ContainerDied","Data":"395648b038f0b43aeee8938954d7a04fd7241a9e1e9a87d4e89f5ab372b85dad"}
Oct 30 20:04:31.164749 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:31.164726    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d5c320-688gn"
Oct 30 20:04:31.296849 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:31.296786    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b73e696-124a-4cab-bb04-442487660026-kube-api-access-lqmvr\" (UniqueName: \"kubernetes.io/projected/0b73e696-124a-4cab-bb04-442487660026-kube-api-access-lqmvr\") pod \"0b73e696-124a-4cab-bb04-442487660026\" (UID: \"0b73e696-124a-4cab-bb04-442487660026\") "
Oct 30 20:04:31.296849 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:31.296823    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b73e696-124a-4cab-bb04-442487660026-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b73e696-124a-4cab-bb04-442487660026-maas-api-openshift-ca\") pod \"0b73e696-124a-4cab-bb04-442487660026\" (UID: \"0b73e696-124a-4cab-bb04-442487660026\") "
Oct 30 20:04:31.297006 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:31.296969    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b73e696-124a-4cab-bb04-442487660026-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b73e696-124a-4cab-bb04-442487660026-maas-api-xks-ca\") pod \"0b73e696-124a-4cab-bb04-442487660026\" (UID: \"0b73e696-124a-4cab-bb04-442487660026\") "
Oct 30 20:04:31.297283 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:31.297251    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b73e696-124a-4cab-bb04-442487660026-maas-api-openshift-ca" pod "0b73e696-124a-4cab-bb04-442487660026" (UID: "0b73e696-124a-4cab-bb04-442487660026"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:31.297433 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:31.297408    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b73e696-124a-4cab-bb04-442487660026-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:31.299076 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:31.299049    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b73e696-124a-4cab-bb04-442487660026-kube-api-access-lqmvr" pod "0b73e696-124a-4cab-bb04-442487660026" (UID: "0b73e696-124a-4cab-bb04-442487660026"). InnerVolumeSpecName "kube-api-access-lqmvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:31.299173 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:31.299132    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b73e696-124a-4cab-bb04-442487660026-maas-api-xks-ca" pod "0b73e696-124a-4cab-bb04-442487660026" (UID: "0b73e696-124a-4cab-bb04-442487660026"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:04:31.398587 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:31.398560    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqmvr\" (UniqueName: \"kubernetes.io/projected/0b73e696-124a-4cab-bb04-442487660026-kube-api-access-lqmvr\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:31.398753 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:31.398737    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b73e696-124a-4cab-bb04-442487660026-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:32.035945 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:32.035914    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d5c320-688gn" event={"ID":"0b73e696-124a-4cab-bb04-442487660026","Type":"ContainerDied","Data":"3041fe007fffd7fcbd6031189c4292916bb8c0f56eee1024569c347540784bbb"}
Oct 30 20:04:32.036113 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:32.035960    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3041fe007fffd7fcbd6031189c4292916bb8c0f56eee1024569c347540784bbb"
Oct 30 20:04:32.036113 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:32.035926    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d5c320-688gn"
Oct 30 20:04:36.605921 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:04:36.605877    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:04:39.297816 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.297778    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"]
Oct 30 20:04:39.298587 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.298565    5958 cpu_manager.go:414] "containerMap: removing container" podUID="0b73e696-124a-4cab-bb04-442487660026" containerName="revoke-keys"
Oct 30 20:04:39.298587 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.298587    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b73e696-124a-4cab-bb04-442487660026" containerName="revoke-keys"
Oct 30 20:04:39.298805 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.298791    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b73e696-124a-4cab-bb04-442487660026" containerName="revoke-keys"
Oct 30 20:04:39.302341 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.302319    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:04:39.340081 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.340035    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-tmp-dir\") pod \"distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8\" (UID: \"504cc4c3-8577-43e6-8feb-5263e0d7d310\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:04:39.340252 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.340160    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-dshm\") pod \"distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8\" (UID: \"504cc4c3-8577-43e6-8feb-5263e0d7d310\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:04:39.340252 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.340208    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-model-cache\") pod \"distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8\" (UID: \"504cc4c3-8577-43e6-8feb-5263e0d7d310\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:04:39.340383 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.340329    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-home\") pod \"distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8\" (UID: \"504cc4c3-8577-43e6-8feb-5263e0d7d310\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:04:39.340383 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.340357    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxssw\" (UniqueName: \"kubernetes.io/projected/504cc4c3-8577-43e6-8feb-5263e0d7d310-kube-api-access-mxssw\") pod \"distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8\" (UID: \"504cc4c3-8577-43e6-8feb-5263e0d7d310\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:04:39.340502 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.340433    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/504cc4c3-8577-43e6-8feb-5263e0d7d310-tls-certs\") pod \"distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8\" (UID: \"504cc4c3-8577-43e6-8feb-5263e0d7d310\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:04:39.408854 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.408824    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-3ccc9f\"/\"openshift-service-ca.crt\""
Oct 30 20:04:39.409042 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.408874    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3ccc9f\"/\"distinct-w1-3ccc9f-kserve-self-signed-certs\""
Oct 30 20:04:39.409042 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.408933    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3ccc9f\"/\"default-dockercfg-z9mnr\""
Oct 30 20:04:39.409644 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.409619    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-3ccc9f\"/\"kube-root-ca.crt\""
Oct 30 20:04:39.411680 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.411656    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"]
Oct 30 20:04:39.441434 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.441405    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-dshm\") pod \"distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8\" (UID: \"504cc4c3-8577-43e6-8feb-5263e0d7d310\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:04:39.441605 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.441511    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-model-cache\") pod \"distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8\" (UID: \"504cc4c3-8577-43e6-8feb-5263e0d7d310\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:04:39.441605 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.441595    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-home\") pod \"distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8\" (UID: \"504cc4c3-8577-43e6-8feb-5263e0d7d310\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:04:39.441732 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.441637    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxssw\" (UniqueName: \"kubernetes.io/projected/504cc4c3-8577-43e6-8feb-5263e0d7d310-kube-api-access-mxssw\") pod \"distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8\" (UID: \"504cc4c3-8577-43e6-8feb-5263e0d7d310\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:04:39.441732 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.441681    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/504cc4c3-8577-43e6-8feb-5263e0d7d310-tls-certs\") pod \"distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8\" (UID: \"504cc4c3-8577-43e6-8feb-5263e0d7d310\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:04:39.441834 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.441784    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-tmp-dir\") pod \"distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8\" (UID: \"504cc4c3-8577-43e6-8feb-5263e0d7d310\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:04:39.442151 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.442129    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-home\") pod \"distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8\" (UID: \"504cc4c3-8577-43e6-8feb-5263e0d7d310\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:04:39.442244 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.442162    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-tmp-dir\") pod \"distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8\" (UID: \"504cc4c3-8577-43e6-8feb-5263e0d7d310\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:04:39.442681 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.442649    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-model-cache\") pod \"distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8\" (UID: \"504cc4c3-8577-43e6-8feb-5263e0d7d310\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:04:39.448337 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.448285    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-dshm\") pod \"distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8\" (UID: \"504cc4c3-8577-43e6-8feb-5263e0d7d310\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:04:39.453778 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.453748    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/504cc4c3-8577-43e6-8feb-5263e0d7d310-tls-certs\") pod \"distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8\" (UID: \"504cc4c3-8577-43e6-8feb-5263e0d7d310\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:04:39.507792 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.507762    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxssw\" (UniqueName: \"kubernetes.io/projected/504cc4c3-8577-43e6-8feb-5263e0d7d310-kube-api-access-mxssw\") pod \"distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8\" (UID: \"504cc4c3-8577-43e6-8feb-5263e0d7d310\") " pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:04:39.614921 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.614891    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:04:39.990620 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:39.990591    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"]
Oct 30 20:04:39.992275 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:04:39.992249    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod504cc4c3_8577_43e6_8feb_5263e0d7d310.slice/crio-9b310dab574aa6c5faa4ff86167bcd91e5a15cfbcbce9446880f9ddf20583e2b WatchSource:0}: Error finding container 9b310dab574aa6c5faa4ff86167bcd91e5a15cfbcbce9446880f9ddf20583e2b: Status 404 returned error can't find the container with id 9b310dab574aa6c5faa4ff86167bcd91e5a15cfbcbce9446880f9ddf20583e2b
Oct 30 20:04:40.099526 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:40.099486    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" event={"ID":"504cc4c3-8577-43e6-8feb-5263e0d7d310","Type":"ContainerStarted","Data":"788f9157e90aaf7b69db4bc12c41735e449a0be4da9d466830e776acb8138453"}
Oct 30 20:04:40.099683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:40.099540    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" event={"ID":"504cc4c3-8577-43e6-8feb-5263e0d7d310","Type":"ContainerStarted","Data":"9b310dab574aa6c5faa4ff86167bcd91e5a15cfbcbce9446880f9ddf20583e2b"}
Oct 30 20:04:40.226071 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:40.226010    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podStartSLOduration=1.225994367 podStartE2EDuration="1.225994367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:40.22168337 +0000 UTC m=+1318.953898759" watchObservedRunningTime="2026-10-30 20:04:40.225994367 +0000 UTC m=+1318.958209816"
Oct 30 20:04:40.616063 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:40.616022    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:04:49.615555 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:49.615515    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:04:49.628229 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:49.628204    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:04:50.173371 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:04:50.173337    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:05:01.566077 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.566041    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"]
Oct 30 20:05:01.570173 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.570156    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.621718 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.621686    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.621905 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.621802    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.621905 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.621839    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a7169d8-4a5a-486d-840c-20a92d74bef5-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.622041 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.621985    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.622102 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.622074    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.622162 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.622136    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.622219 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.622202    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.622316 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.622286    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.622519 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.622498    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7csq\" (UniqueName: \"kubernetes.io/projected/0a7169d8-4a5a-486d-840c-20a92d74bef5-kube-api-access-z7csq\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.677093 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.677046    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"]
Oct 30 20:05:01.724213 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.724180    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.724392 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.724244    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.724392 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.724275    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.724392 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.724325    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.724392 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.724381    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.724646 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.724411    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7csq\" (UniqueName: \"kubernetes.io/projected/0a7169d8-4a5a-486d-840c-20a92d74bef5-kube-api-access-z7csq\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.724704 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.724654    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.724704 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.724671    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.724871 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.724850    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.724948 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.724935    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.724985 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.724965    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a7169d8-4a5a-486d-840c-20a92d74bef5-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.725299 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.725279    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.725513 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.725488    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.725679 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.725662    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a7169d8-4a5a-486d-840c-20a92d74bef5-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.726836 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.726817    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.726942 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.726923    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.787441 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.787409    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.787763 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.787746    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7csq\" (UniqueName: \"kubernetes.io/projected/0a7169d8-4a5a-486d-840c-20a92d74bef5-kube-api-access-z7csq\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.818715 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.818662    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"]
Oct 30 20:05:01.824074 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.824055    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:01.884391 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.883999    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:01.888178 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.888154    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"]
Oct 30 20:05:01.927112 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.927081    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:01.927281 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.927173    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:01.927281 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.927204    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:01.927281 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.927221    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:01.927281 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.927239    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:01.927281 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.927270    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:01.927594 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.927291    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:01.927594 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.927435    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:01.927594 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:01.927511    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpwbt\" (UniqueName: \"kubernetes.io/projected/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-kube-api-access-qpwbt\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:02.028430 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.028400    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:02.028626 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.028461    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:02.028626 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.028490    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:02.028626 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.028506    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:02.028626 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.028526    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:02.028626 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.028564    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:02.028956 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.028808    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:02.028956 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.028894    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:02.028956 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.028940    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:02.029334 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.029291    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpwbt\" (UniqueName: \"kubernetes.io/projected/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-kube-api-access-qpwbt\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:02.029334 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.029320    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:02.030036 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.030010    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:02.030261 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.030242    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:02.030364 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.030340    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:02.031017 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.030994    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:02.031669 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.031649    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:02.090085 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.090037    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:02.090160 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.090115    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpwbt\" (UniqueName: \"kubernetes.io/projected/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-kube-api-access-qpwbt\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:02.139020 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.139000    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:02.329967 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.329941    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"]
Oct 30 20:05:02.331986 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:05:02.331954    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7169d8_4a5a_486d_840c_20a92d74bef5.slice/crio-c8324b8a6b79a3d3f117724cbfc9ea98409ce46e5ed6533ca5d1fbf1249f0288 WatchSource:0}: Error finding container c8324b8a6b79a3d3f117724cbfc9ea98409ce46e5ed6533ca5d1fbf1249f0288: Status 404 returned error can't find the container with id c8324b8a6b79a3d3f117724cbfc9ea98409ce46e5ed6533ca5d1fbf1249f0288
Oct 30 20:05:02.334332 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.334300    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:05:02.334404 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.334364    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:05:02.334404 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.334396    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:05:02.522880 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.522854    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"]
Oct 30 20:05:02.524329 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:05:02.524304    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf382f3fe_b7ff_4e1e_a36b_98b8b0882937.slice/crio-cf8c33d21babc7be4b182c47952f806dbcfcb0ef937f6efbdb8040e22b6f3485 WatchSource:0}: Error finding container cf8c33d21babc7be4b182c47952f806dbcfcb0ef937f6efbdb8040e22b6f3485: Status 404 returned error can't find the container with id cf8c33d21babc7be4b182c47952f806dbcfcb0ef937f6efbdb8040e22b6f3485
Oct 30 20:05:02.527079 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.527045    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:05:02.527174 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.527105    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:05:02.527174 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:02.527136    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:05:03.296839 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:03.296799    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6" event={"ID":"f382f3fe-b7ff-4e1e-a36b-98b8b0882937","Type":"ContainerStarted","Data":"d89cb5e7a852955573bbc24bfdc30e7f2583e24f905dde6a06e793a59659fe23"}
Oct 30 20:05:03.297332 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:03.296858    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6" event={"ID":"f382f3fe-b7ff-4e1e-a36b-98b8b0882937","Type":"ContainerStarted","Data":"cf8c33d21babc7be4b182c47952f806dbcfcb0ef937f6efbdb8040e22b6f3485"}
Oct 30 20:05:03.299591 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:03.299560    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq" event={"ID":"0a7169d8-4a5a-486d-840c-20a92d74bef5","Type":"ContainerStarted","Data":"12b1e92d1e057f26118e0185e8988067370b9a8c83fa962073a8d7677cd5aef7"}
Oct 30 20:05:03.299741 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:03.299604    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq" event={"ID":"0a7169d8-4a5a-486d-840c-20a92d74bef5","Type":"ContainerStarted","Data":"c8324b8a6b79a3d3f117724cbfc9ea98409ce46e5ed6533ca5d1fbf1249f0288"}
Oct 30 20:05:03.430391 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:03.430187    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6" podStartSLOduration=2.429901534 podStartE2EDuration="2.429901534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:03.425579246 +0000 UTC m=+1342.157794636" watchObservedRunningTime="2026-10-30 20:05:03.429901534 +0000 UTC m=+1342.162116917"
Oct 30 20:05:03.889237 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:03.887926    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:03.892664 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:03.892626    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 20:05:03.892804 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:03.892727    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq" podUID="0a7169d8-4a5a-486d-840c-20a92d74bef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 20:05:04.146544 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:04.139781    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:04.168477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:04.150570    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 20:05:04.168477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:04.150636    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6" podUID="f382f3fe-b7ff-4e1e-a36b-98b8b0882937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 20:05:04.885349 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:04.884898    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 20:05:04.885349 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:04.884961    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq" podUID="0a7169d8-4a5a-486d-840c-20a92d74bef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 20:05:05.144538 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:05.139890    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 20:05:05.144538 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:05.139954    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6" podUID="f382f3fe-b7ff-4e1e-a36b-98b8b0882937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 20:05:05.886541 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:05.885832    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 20:05:05.886541 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:05.885901    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq" podUID="0a7169d8-4a5a-486d-840c-20a92d74bef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 20:05:06.172535 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:06.168552    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 20:05:06.172535 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:06.168611    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6" podUID="f382f3fe-b7ff-4e1e-a36b-98b8b0882937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 20:05:06.611341 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:05:06.611298    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:05:06.892646 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:06.892549    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 20:05:06.893195 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:06.893163    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq" podUID="0a7169d8-4a5a-486d-840c-20a92d74bef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 20:05:07.152070 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:07.151976    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 20:05:07.170392 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:07.170351    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6" podUID="f382f3fe-b7ff-4e1e-a36b-98b8b0882937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 20:05:07.886068 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:07.885883    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 20:05:07.886068 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:07.885949    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq" podUID="0a7169d8-4a5a-486d-840c-20a92d74bef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 20:05:08.143541 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:08.139869    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 20:05:08.143541 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:08.139935    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6" podUID="f382f3fe-b7ff-4e1e-a36b-98b8b0882937" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 20:05:08.884646 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:08.884602    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 20:05:08.884849 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:08.884690    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq" podUID="0a7169d8-4a5a-486d-840c-20a92d74bef5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 20:05:09.209045 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:09.208958    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:09.209655 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:09.209637    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:09.210368 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:09.210342    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:05:09.337416 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:09.337350    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq" podStartSLOduration=8.3373303 podStartE2EDuration="8.3373303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:03.573763418 +0000 UTC m=+1342.305978808" watchObservedRunningTime="2026-10-30 20:05:09.3373303 +0000 UTC m=+1348.069545691"
Oct 30 20:05:09.888353 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:09.888322    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:09.888831 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:09.888819    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:09.889674 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:09.889652    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:05:13.601633 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:13.601595    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"]
Oct 30 20:05:13.606133 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:13.606114    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:05:13.710371 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:13.710336    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bebb42\"/\"distinct-2-w4-bebb42-kserve-self-signed-certs\""
Oct 30 20:05:13.713273 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:13.713243    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"]
Oct 30 20:05:13.753052 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:13.753013    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b0c566-35a5-41b8-b853-0524039ce73a-tls-certs\") pod \"distinct-2-w4-bebb42-kserve-797567597-jdv25\" (UID: \"68b0c566-35a5-41b8-b853-0524039ce73a\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:05:13.753239 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:13.753166    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-model-cache\") pod \"distinct-2-w4-bebb42-kserve-797567597-jdv25\" (UID: \"68b0c566-35a5-41b8-b853-0524039ce73a\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:05:13.753293 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:13.753233    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-tmp-dir\") pod \"distinct-2-w4-bebb42-kserve-797567597-jdv25\" (UID: \"68b0c566-35a5-41b8-b853-0524039ce73a\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:05:13.753365 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:13.753349    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-home\") pod \"distinct-2-w4-bebb42-kserve-797567597-jdv25\" (UID: \"68b0c566-35a5-41b8-b853-0524039ce73a\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:05:13.753417 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:13.753406    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-dshm\") pod \"distinct-2-w4-bebb42-kserve-797567597-jdv25\" (UID: \"68b0c566-35a5-41b8-b853-0524039ce73a\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:05:13.753506 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:13.753493    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lsp2\" (UniqueName: \"kubernetes.io/projected/68b0c566-35a5-41b8-b853-0524039ce73a-kube-api-access-8lsp2\") pod \"distinct-2-w4-bebb42-kserve-797567597-jdv25\" (UID: \"68b0c566-35a5-41b8-b853-0524039ce73a\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:05:13.854507 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:13.854409    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-home\") pod \"distinct-2-w4-bebb42-kserve-797567597-jdv25\" (UID: \"68b0c566-35a5-41b8-b853-0524039ce73a\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:05:13.854668 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:13.854520    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-dshm\") pod \"distinct-2-w4-bebb42-kserve-797567597-jdv25\" (UID: \"68b0c566-35a5-41b8-b853-0524039ce73a\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:05:13.854668 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:13.854557    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lsp2\" (UniqueName: \"kubernetes.io/projected/68b0c566-35a5-41b8-b853-0524039ce73a-kube-api-access-8lsp2\") pod \"distinct-2-w4-bebb42-kserve-797567597-jdv25\" (UID: \"68b0c566-35a5-41b8-b853-0524039ce73a\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:05:13.854775 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:13.854679    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b0c566-35a5-41b8-b853-0524039ce73a-tls-certs\") pod \"distinct-2-w4-bebb42-kserve-797567597-jdv25\" (UID: \"68b0c566-35a5-41b8-b853-0524039ce73a\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:05:13.854775 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:13.854758    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-model-cache\") pod \"distinct-2-w4-bebb42-kserve-797567597-jdv25\" (UID: \"68b0c566-35a5-41b8-b853-0524039ce73a\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:05:13.854875 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:13.854809    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-tmp-dir\") pod \"distinct-2-w4-bebb42-kserve-797567597-jdv25\" (UID: \"68b0c566-35a5-41b8-b853-0524039ce73a\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:05:13.854875 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:13.854827    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-home\") pod \"distinct-2-w4-bebb42-kserve-797567597-jdv25\" (UID: \"68b0c566-35a5-41b8-b853-0524039ce73a\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:05:13.855121 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:13.855095    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-model-cache\") pod \"distinct-2-w4-bebb42-kserve-797567597-jdv25\" (UID: \"68b0c566-35a5-41b8-b853-0524039ce73a\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:05:13.855299 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:13.855280    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-tmp-dir\") pod \"distinct-2-w4-bebb42-kserve-797567597-jdv25\" (UID: \"68b0c566-35a5-41b8-b853-0524039ce73a\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:05:13.856882 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:13.856858    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-dshm\") pod \"distinct-2-w4-bebb42-kserve-797567597-jdv25\" (UID: \"68b0c566-35a5-41b8-b853-0524039ce73a\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:05:13.857166 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:13.857145    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b0c566-35a5-41b8-b853-0524039ce73a-tls-certs\") pod \"distinct-2-w4-bebb42-kserve-797567597-jdv25\" (UID: \"68b0c566-35a5-41b8-b853-0524039ce73a\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:05:13.916787 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:13.916749    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lsp2\" (UniqueName: \"kubernetes.io/projected/68b0c566-35a5-41b8-b853-0524039ce73a-kube-api-access-8lsp2\") pod \"distinct-2-w4-bebb42-kserve-797567597-jdv25\" (UID: \"68b0c566-35a5-41b8-b853-0524039ce73a\") " pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:05:13.919678 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:13.919655    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:05:14.299399 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:14.299371    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"]
Oct 30 20:05:14.300961 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:05:14.300933    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68b0c566_35a5_41b8_b853_0524039ce73a.slice/crio-67450dccc31e7d6ad659c1a66998a03adba1cc237d579891c13dbba946d9dae3 WatchSource:0}: Error finding container 67450dccc31e7d6ad659c1a66998a03adba1cc237d579891c13dbba946d9dae3: Status 404 returned error can't find the container with id 67450dccc31e7d6ad659c1a66998a03adba1cc237d579891c13dbba946d9dae3
Oct 30 20:05:14.386188 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:14.386149    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" event={"ID":"68b0c566-35a5-41b8-b853-0524039ce73a","Type":"ContainerStarted","Data":"400a1dc92c4df55ff03dbad03fa4441d5aaa21a8842cf7fcb384dea917599a99"}
Oct 30 20:05:14.386369 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:14.386200    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" event={"ID":"68b0c566-35a5-41b8-b853-0524039ce73a","Type":"ContainerStarted","Data":"67450dccc31e7d6ad659c1a66998a03adba1cc237d579891c13dbba946d9dae3"}
Oct 30 20:05:14.506207 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:14.506138    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" podStartSLOduration=1.5061196209999999 podStartE2EDuration="1.506119621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:14.504654741 +0000 UTC m=+1353.236870130" watchObservedRunningTime="2026-10-30 20:05:14.506119621 +0000 UTC m=+1353.238335013"
Oct 30 20:05:14.920556 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:14.920522    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:05:23.920859 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:23.920759    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:05:23.933677 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:23.933652    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:05:24.457968 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:24.457937    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:05:31.027683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.027618    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"]
Oct 30 20:05:31.031787 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.031767    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.147058 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.147018    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"]
Oct 30 20:05:31.211243 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.211203    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.211401 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.211271    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.211401 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.211339    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.211401 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.211384    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.211401 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.211400    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9ss5\" (UniqueName: \"kubernetes.io/projected/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-kube-api-access-q9ss5\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.211665 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.211553    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.211665 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.211626    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.211780 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.211764    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.211847 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.211819    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.312773 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.312679    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.312773 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.312758    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.313007 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.312794    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.313007 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.312823    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.313007 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.312877    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.313007 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.312901    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9ss5\" (UniqueName: \"kubernetes.io/projected/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-kube-api-access-q9ss5\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.313007 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.312941    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.313007 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.312989    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.313305 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.313097    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.313305 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.313148    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.313305 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.313237    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.313535 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.313337    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.313535 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.313488    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.314054 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.314032    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.315575 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.315554    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.315760 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.315739    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.375260 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.375226    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.375509 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.375485    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9ss5\" (UniqueName: \"kubernetes.io/projected/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-kube-api-access-q9ss5\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:31.645419 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:31.645384    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:32.076281 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:32.076255    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"]
Oct 30 20:05:32.077581 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:05:32.077557    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a80c44_b6f8_44bc_9a7d_86a43f48f7c7.slice/crio-28085f9cd55b62c34f3c25240673699b9b114053bd3c8e9ab75d96731eb17a54 WatchSource:0}: Error finding container 28085f9cd55b62c34f3c25240673699b9b114053bd3c8e9ab75d96731eb17a54: Status 404 returned error can't find the container with id 28085f9cd55b62c34f3c25240673699b9b114053bd3c8e9ab75d96731eb17a54
Oct 30 20:05:32.079923 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:32.079884    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:05:32.080036 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:32.079954    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:05:32.080036 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:32.079985    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:05:32.506268 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:32.505825    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv" event={"ID":"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7","Type":"ContainerStarted","Data":"037fa331b2b7055496b4a412573cedb6bb8b548e78af4530d27cd1a203f583dd"}
Oct 30 20:05:32.506268 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:32.505891    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv" event={"ID":"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7","Type":"ContainerStarted","Data":"28085f9cd55b62c34f3c25240673699b9b114053bd3c8e9ab75d96731eb17a54"}
Oct 30 20:05:32.654476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:32.646571    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:32.681233 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:32.681175    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv" podStartSLOduration=2.681161505 podStartE2EDuration="2.681161505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:32.677258376 +0000 UTC m=+1371.409473768" watchObservedRunningTime="2026-10-30 20:05:32.681161505 +0000 UTC m=+1371.413376895"
Oct 30 20:05:33.651632 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:33.651592    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 20:05:33.652211 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:33.652179    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv" podUID="54a80c44-b6f8-44bc-9a7d-86a43f48f7c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 20:05:34.656476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:34.656309    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 20:05:34.656476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:34.656371    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv" podUID="54a80c44-b6f8-44bc-9a7d-86a43f48f7c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 20:05:35.655476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:35.653901    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 20:05:35.655476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:35.653967    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv" podUID="54a80c44-b6f8-44bc-9a7d-86a43f48f7c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 20:05:36.606106 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:05:36.606068    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:05:36.658477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:36.653884    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 20:05:36.658477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:36.653950    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv" podUID="54a80c44-b6f8-44bc-9a7d-86a43f48f7c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 20:05:37.655476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:37.647864    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 20:05:37.655476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:37.647928    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv" podUID="54a80c44-b6f8-44bc-9a7d-86a43f48f7c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 20:05:39.008780 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:39.008690    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:39.009471 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:39.009431    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:39.010390 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:39.010368    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:05:57.900722 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:57.900683    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"]
Oct 30 20:05:57.904802 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:57.904782    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:05:57.942419 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:57.942382    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-model-cache\") pod \"distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb\" (UID: \"bf720d46-c112-42e5-8a6b-11c6cb96042d\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:05:57.942656 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:57.942465    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-home\") pod \"distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb\" (UID: \"bf720d46-c112-42e5-8a6b-11c6cb96042d\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:05:57.942656 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:57.942516    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-tmp-dir\") pod \"distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb\" (UID: \"bf720d46-c112-42e5-8a6b-11c6cb96042d\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:05:57.942795 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:57.942764    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-dshm\") pod \"distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb\" (UID: \"bf720d46-c112-42e5-8a6b-11c6cb96042d\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:05:57.942915 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:57.942895    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtwqw\" (UniqueName: \"kubernetes.io/projected/bf720d46-c112-42e5-8a6b-11c6cb96042d-kube-api-access-wtwqw\") pod \"distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb\" (UID: \"bf720d46-c112-42e5-8a6b-11c6cb96042d\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:05:57.942977 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:57.942934    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf720d46-c112-42e5-8a6b-11c6cb96042d-tls-certs\") pod \"distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb\" (UID: \"bf720d46-c112-42e5-8a6b-11c6cb96042d\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:05:57.997711 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:57.997675    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d67ea9\"/\"default-dockercfg-q5s7s\""
Oct 30 20:05:58.010579 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:58.010550    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d67ea9\"/\"distinct-w2-d67ea9-kserve-self-signed-certs\""
Oct 30 20:05:58.010755 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:58.010681    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d67ea9\"/\"kube-root-ca.crt\""
Oct 30 20:05:58.013669 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:58.013644    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"]
Oct 30 20:05:58.031760 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:58.031726    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d67ea9\"/\"openshift-service-ca.crt\""
Oct 30 20:05:58.044398 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:58.044361    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-model-cache\") pod \"distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb\" (UID: \"bf720d46-c112-42e5-8a6b-11c6cb96042d\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:05:58.044680 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:58.044424    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-home\") pod \"distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb\" (UID: \"bf720d46-c112-42e5-8a6b-11c6cb96042d\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:05:58.044680 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:58.044479    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-tmp-dir\") pod \"distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb\" (UID: \"bf720d46-c112-42e5-8a6b-11c6cb96042d\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:05:58.044680 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:58.044629    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-dshm\") pod \"distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb\" (UID: \"bf720d46-c112-42e5-8a6b-11c6cb96042d\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:05:58.044867 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:58.044716    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtwqw\" (UniqueName: \"kubernetes.io/projected/bf720d46-c112-42e5-8a6b-11c6cb96042d-kube-api-access-wtwqw\") pod \"distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb\" (UID: \"bf720d46-c112-42e5-8a6b-11c6cb96042d\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:05:58.044867 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:58.044749    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf720d46-c112-42e5-8a6b-11c6cb96042d-tls-certs\") pod \"distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb\" (UID: \"bf720d46-c112-42e5-8a6b-11c6cb96042d\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:05:58.044867 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:58.044794    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-model-cache\") pod \"distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb\" (UID: \"bf720d46-c112-42e5-8a6b-11c6cb96042d\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:05:58.044867 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:58.044849    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-home\") pod \"distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb\" (UID: \"bf720d46-c112-42e5-8a6b-11c6cb96042d\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:05:58.045076 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:58.044906    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-tmp-dir\") pod \"distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb\" (UID: \"bf720d46-c112-42e5-8a6b-11c6cb96042d\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:05:58.047009 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:58.046984    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-dshm\") pod \"distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb\" (UID: \"bf720d46-c112-42e5-8a6b-11c6cb96042d\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:05:58.047359 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:58.047338    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf720d46-c112-42e5-8a6b-11c6cb96042d-tls-certs\") pod \"distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb\" (UID: \"bf720d46-c112-42e5-8a6b-11c6cb96042d\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:05:58.119637 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:58.119604    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtwqw\" (UniqueName: \"kubernetes.io/projected/bf720d46-c112-42e5-8a6b-11c6cb96042d-kube-api-access-wtwqw\") pod \"distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb\" (UID: \"bf720d46-c112-42e5-8a6b-11c6cb96042d\") " pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:05:58.218906 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:58.218821    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:05:58.619004 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:58.618960    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"]
Oct 30 20:05:58.619440 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:05:58.619414    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf720d46_c112_42e5_8a6b_11c6cb96042d.slice/crio-a3501669b3a02645d50946f0531b5feffefaeacab00ee83b7774ad1d03d1b105 WatchSource:0}: Error finding container a3501669b3a02645d50946f0531b5feffefaeacab00ee83b7774ad1d03d1b105: Status 404 returned error can't find the container with id a3501669b3a02645d50946f0531b5feffefaeacab00ee83b7774ad1d03d1b105
Oct 30 20:05:58.756023 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:58.755990    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" event={"ID":"bf720d46-c112-42e5-8a6b-11c6cb96042d","Type":"ContainerStarted","Data":"80855c98ca95a234482722c9e14a075988a01af2bab1918f541d3a01b0b8e254"}
Oct 30 20:05:58.756216 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:58.756030    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" event={"ID":"bf720d46-c112-42e5-8a6b-11c6cb96042d","Type":"ContainerStarted","Data":"a3501669b3a02645d50946f0531b5feffefaeacab00ee83b7774ad1d03d1b105"}
Oct 30 20:05:58.885600 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:58.885499    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podStartSLOduration=1.8854857250000001 podStartE2EDuration="1.885485725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:58.884096108 +0000 UTC m=+1397.616311497" watchObservedRunningTime="2026-10-30 20:05:58.885485725 +0000 UTC m=+1397.617701114"
Oct 30 20:05:59.219307 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:05:59.219231    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:06:06.610488 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:06:06.610425    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:06:08.219529 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:08.219497    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:06:08.231489 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:08.231440    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:06:08.835049 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:08.835016    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:06:16.763482 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.763431    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"]
Oct 30 20:06:16.768966 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.768944    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.793116 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.793082    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.793253 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.793182    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a977973c-341c-48ec-9df5-8035da148b90-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.793253 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.793220    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.793253 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.793242    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.793480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.793427    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.793588 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.793571    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.793647 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.793636    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xjb4\" (UniqueName: \"kubernetes.io/projected/a977973c-341c-48ec-9df5-8035da148b90-kube-api-access-5xjb4\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.793700 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.793667    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a977973c-341c-48ec-9df5-8035da148b90-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.793791 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.793723    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a977973c-341c-48ec-9df5-8035da148b90-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.886116 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.886071    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"]
Oct 30 20:06:16.895616 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.895582    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xjb4\" (UniqueName: \"kubernetes.io/projected/a977973c-341c-48ec-9df5-8035da148b90-kube-api-access-5xjb4\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.895761 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.895629    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a977973c-341c-48ec-9df5-8035da148b90-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.895761 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.895667    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a977973c-341c-48ec-9df5-8035da148b90-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.895761 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.895757    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.895925 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.895800    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a977973c-341c-48ec-9df5-8035da148b90-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.895925 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.895878    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.896028 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.895923    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.896028 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.896018    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.896163 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.896141    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.896254 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.896227    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.896414 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.896392    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.896531 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.896468    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.896598 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.896527    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a977973c-341c-48ec-9df5-8035da148b90-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.896598 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.896561    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.898860 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.898834    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.899716 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.899693    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a977973c-341c-48ec-9df5-8035da148b90-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.957269 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.957238    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a977973c-341c-48ec-9df5-8035da148b90-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:16.957574 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:16.957551    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xjb4\" (UniqueName: \"kubernetes.io/projected/a977973c-341c-48ec-9df5-8035da148b90-kube-api-access-5xjb4\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:17.085119 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.085089    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:17.173372 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.172642    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"]
Oct 30 20:06:17.178176 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.178079    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.200980 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.200943    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.201133 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.201088    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.201204 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.201131    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.201204 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.201157    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.201319 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.201277    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.201373 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.201333    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.201428 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.201382    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.201428 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.201405    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.201768 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.201744    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st82h\" (UniqueName: \"kubernetes.io/projected/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-kube-api-access-st82h\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.257895 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.257865    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"]
Oct 30 20:06:17.303061 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.303032    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.303963 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.303934    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.304089 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.303980    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.304163 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.304104    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.304415 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.304395    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-workload-certs\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.304523 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.304498    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.304595 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.304546    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.304796 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.304778    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-credential-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.304988 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.304968    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-workload-socket\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.305061 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.304571    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.305256 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.305239    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-data\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.305343 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.305324    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st82h\" (UniqueName: \"kubernetes.io/projected/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-kube-api-access-st82h\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.305735 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.305711    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.306053 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.306034    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-envoy\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.306516 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.306495    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istiod-ca-cert\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.307983 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.307962    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-podinfo\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.367086 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.367006    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-token\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.368846 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.368824    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st82h\" (UniqueName: \"kubernetes.io/projected/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-kube-api-access-st82h\") pod \"e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") " pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.496027 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.495970    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:17.529028 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.526944    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"]
Oct 30 20:06:17.533176 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:06:17.533151    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda977973c_341c_48ec_9df5_8035da148b90.slice/crio-0c6bff43f51631384279a5eeade6815013ad2ba24e4b67e0cffda5b2c41d0273 WatchSource:0}: Error finding container 0c6bff43f51631384279a5eeade6815013ad2ba24e4b67e0cffda5b2c41d0273: Status 404 returned error can't find the container with id 0c6bff43f51631384279a5eeade6815013ad2ba24e4b67e0cffda5b2c41d0273
Oct 30 20:06:17.535832 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.535783    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:06:17.535952 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.535909    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:06:17.536009 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.535950    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:06:17.893039 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.892929    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m" event={"ID":"a977973c-341c-48ec-9df5-8035da148b90","Type":"ContainerStarted","Data":"956d556026f44b19e237d54461eea3f29e9376fab24aa4aad16d5348f55964ff"}
Oct 30 20:06:17.893039 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:17.893005    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m" event={"ID":"a977973c-341c-48ec-9df5-8035da148b90","Type":"ContainerStarted","Data":"0c6bff43f51631384279a5eeade6815013ad2ba24e4b67e0cffda5b2c41d0273"}
Oct 30 20:06:18.075481 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:18.067237    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"]
Oct 30 20:06:18.092164 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:18.092129    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:18.102596 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:18.102561    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 20:06:18.102856 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:18.102825    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m" podUID="a977973c-341c-48ec-9df5-8035da148b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 20:06:18.129477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:18.112375    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m" podStartSLOduration=2.112355556 podStartE2EDuration="2.112355556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:18.091831235 +0000 UTC m=+1416.824046626" watchObservedRunningTime="2026-10-30 20:06:18.112355556 +0000 UTC m=+1416.844570947"
Oct 30 20:06:18.145530 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:18.135051    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:06:18.145530 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:18.135149    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:06:18.148003 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:18.147959    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:06:18.947484 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:18.939117    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb" event={"ID":"e6fa742a-7bee-4a79-b5a1-aa33c24552f6","Type":"ContainerStarted","Data":"eceece4314ddfd5c3a2b586fbb7ca992cebbcdebb601165e55dd0d936d024fa0"}
Oct 30 20:06:18.948291 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:18.948259    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb" event={"ID":"e6fa742a-7bee-4a79-b5a1-aa33c24552f6","Type":"ContainerStarted","Data":"0474177e2009796899e0302f1d65c2bdb6ab593bc04717ef407a566c0186423f"}
Oct 30 20:06:19.091200 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:19.089830    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 20:06:19.091200 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:19.089891    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m" podUID="a977973c-341c-48ec-9df5-8035da148b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 20:06:19.097515 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:19.097429    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb" podStartSLOduration=2.097407147 podStartE2EDuration="2.097407147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:19.084513834 +0000 UTC m=+1417.816729225" watchObservedRunningTime="2026-10-30 20:06:19.097407147 +0000 UTC m=+1417.829622538"
Oct 30 20:06:19.500340 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:19.500304    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:19.500640 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:19.500549    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 20:06:19.500820 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:19.500795    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb" podUID="e6fa742a-7bee-4a79-b5a1-aa33c24552f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 20:06:19.768672 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:19.768543    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" podUID="6e20c97b-66ce-485b-ba8d-90c2d9e1fa76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:19.825476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:19.825076    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:19.837924 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:19.837583    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:19.993297 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:19.990376    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:20.053534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:20.017536    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:20.312536 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:20.309559    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:20.322592 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:20.322466    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 20:06:20.322592 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:20.322523    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:20.322592 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:20.322523    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m" podUID="a977973c-341c-48ec-9df5-8035da148b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 20:06:20.329648 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:20.329616    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb1e9517-jmwhb"]
Oct 30 20:06:20.434807 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:20.434773    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb1e9517-jmwhb"]
Oct 30 20:06:20.434999 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:20.434955    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1e9517-jmwhb"
Oct 30 20:06:20.497479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:20.497427    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 20:06:20.497706 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:20.497676    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb" podUID="e6fa742a-7bee-4a79-b5a1-aa33c24552f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 20:06:20.508705 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:20.508654    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:06:20.605291 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:20.604703    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aac6c75-c3d7-4df7-8b56-f08a89dffe09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb1e9517-jmwhb\" (UID: \"9aac6c75-c3d7-4df7-8b56-f08a89dffe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1e9517-jmwhb"
Oct 30 20:06:20.605291 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:20.604830    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aac6c75-c3d7-4df7-8b56-f08a89dffe09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb1e9517-jmwhb\" (UID: \"9aac6c75-c3d7-4df7-8b56-f08a89dffe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1e9517-jmwhb"
Oct 30 20:06:20.605291 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:20.605255    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jhfr\" (UniqueName: \"kubernetes.io/projected/9aac6c75-c3d7-4df7-8b56-f08a89dffe09-kube-api-access-6jhfr\") pod \"maas-api-revoke-sub-fb1e9517-jmwhb\" (UID: \"9aac6c75-c3d7-4df7-8b56-f08a89dffe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1e9517-jmwhb"
Oct 30 20:06:20.608902 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:20.608883    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:06:20.706999 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:20.706552    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aac6c75-c3d7-4df7-8b56-f08a89dffe09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb1e9517-jmwhb\" (UID: \"9aac6c75-c3d7-4df7-8b56-f08a89dffe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1e9517-jmwhb"
Oct 30 20:06:20.706999 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:20.706630    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aac6c75-c3d7-4df7-8b56-f08a89dffe09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb1e9517-jmwhb\" (UID: \"9aac6c75-c3d7-4df7-8b56-f08a89dffe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1e9517-jmwhb"
Oct 30 20:06:20.706999 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:20.706822    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jhfr\" (UniqueName: \"kubernetes.io/projected/9aac6c75-c3d7-4df7-8b56-f08a89dffe09-kube-api-access-6jhfr\") pod \"maas-api-revoke-sub-fb1e9517-jmwhb\" (UID: \"9aac6c75-c3d7-4df7-8b56-f08a89dffe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1e9517-jmwhb"
Oct 30 20:06:20.707328 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:20.707303    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aac6c75-c3d7-4df7-8b56-f08a89dffe09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb1e9517-jmwhb\" (UID: \"9aac6c75-c3d7-4df7-8b56-f08a89dffe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1e9517-jmwhb"
Oct 30 20:06:20.709313 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:20.709287    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aac6c75-c3d7-4df7-8b56-f08a89dffe09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb1e9517-jmwhb\" (UID: \"9aac6c75-c3d7-4df7-8b56-f08a89dffe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1e9517-jmwhb"
Oct 30 20:06:20.796474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:20.794541    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jhfr\" (UniqueName: \"kubernetes.io/projected/9aac6c75-c3d7-4df7-8b56-f08a89dffe09-kube-api-access-6jhfr\") pod \"maas-api-revoke-sub-fb1e9517-jmwhb\" (UID: \"9aac6c75-c3d7-4df7-8b56-f08a89dffe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1e9517-jmwhb"
Oct 30 20:06:20.810497 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:20.809038    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1e9517-jmwhb"
Oct 30 20:06:21.108078 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:21.108036    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 20:06:21.108545 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:21.108102    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m" podUID="a977973c-341c-48ec-9df5-8035da148b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 20:06:21.389493 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:21.375073    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb1e9517-jmwhb"]
Oct 30 20:06:21.498779 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:21.497737    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 20:06:21.498779 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:21.497796    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb" podUID="e6fa742a-7bee-4a79-b5a1-aa33c24552f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 20:06:21.499303 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:21.499269    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1e9517-jmwhb" event={"ID":"9aac6c75-c3d7-4df7-8b56-f08a89dffe09","Type":"ContainerStarted","Data":"7ae04bae960d21e28d2af9e5ae4630fb7c6216b137770dbecaaa6e81248d4f55"}
Oct 30 20:06:22.089483 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:22.088661    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 20:06:22.089483 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:22.088727    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m" podUID="a977973c-341c-48ec-9df5-8035da148b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 20:06:22.496848 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:22.496763    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 20:06:22.496848 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:22.496822    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb" podUID="e6fa742a-7bee-4a79-b5a1-aa33c24552f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 20:06:22.510204 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:22.510107    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1e9517-jmwhb" event={"ID":"9aac6c75-c3d7-4df7-8b56-f08a89dffe09","Type":"ContainerStarted","Data":"93889a1d428e8443a96204cff1951d122b83da348308fbcf933dcef907882518"}
Oct 30 20:06:22.632675 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:22.632608    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1e9517-jmwhb" podStartSLOduration=3.632592399 podStartE2EDuration="3.632592399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:22.629825356 +0000 UTC m=+1421.362040748" watchObservedRunningTime="2026-10-30 20:06:22.632592399 +0000 UTC m=+1421.364807789"
Oct 30 20:06:23.086953 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:23.086908    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 20:06:23.087151 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:23.086975    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m" podUID="a977973c-341c-48ec-9df5-8035da148b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 20:06:23.496658 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:23.496574    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 20:06:23.496658 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:23.496630    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb" podUID="e6fa742a-7bee-4a79-b5a1-aa33c24552f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 20:06:23.518748 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:23.518715    5958 generic.go:356] "Generic (PLEG): container finished" podID="9aac6c75-c3d7-4df7-8b56-f08a89dffe09" containerID="93889a1d428e8443a96204cff1951d122b83da348308fbcf933dcef907882518" exitCode=0
Oct 30 20:06:23.519110 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:23.518802    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1e9517-jmwhb" event={"ID":"9aac6c75-c3d7-4df7-8b56-f08a89dffe09","Type":"ContainerDied","Data":"93889a1d428e8443a96204cff1951d122b83da348308fbcf933dcef907882518"}
Oct 30 20:06:24.085967 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:24.085926    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 20:06:24.086241 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:24.086211    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m" podUID="a977973c-341c-48ec-9df5-8035da148b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 20:06:24.497512 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:24.497378    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 20:06:24.497512 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:24.497437    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb" podUID="e6fa742a-7bee-4a79-b5a1-aa33c24552f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 20:06:24.822784 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:24.822754    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1e9517-jmwhb"
Oct 30 20:06:24.831465 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:24.831432    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aac6c75-c3d7-4df7-8b56-f08a89dffe09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aac6c75-c3d7-4df7-8b56-f08a89dffe09-maas-api-openshift-ca\") pod \"9aac6c75-c3d7-4df7-8b56-f08a89dffe09\" (UID: \"9aac6c75-c3d7-4df7-8b56-f08a89dffe09\") "
Oct 30 20:06:24.831615 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:24.831591    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aac6c75-c3d7-4df7-8b56-f08a89dffe09-kube-api-access-6jhfr\" (UniqueName: \"kubernetes.io/projected/9aac6c75-c3d7-4df7-8b56-f08a89dffe09-kube-api-access-6jhfr\") pod \"9aac6c75-c3d7-4df7-8b56-f08a89dffe09\" (UID: \"9aac6c75-c3d7-4df7-8b56-f08a89dffe09\") "
Oct 30 20:06:24.831684 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:24.831648    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aac6c75-c3d7-4df7-8b56-f08a89dffe09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aac6c75-c3d7-4df7-8b56-f08a89dffe09-maas-api-xks-ca\") pod \"9aac6c75-c3d7-4df7-8b56-f08a89dffe09\" (UID: \"9aac6c75-c3d7-4df7-8b56-f08a89dffe09\") "
Oct 30 20:06:24.831821 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:24.831787    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aac6c75-c3d7-4df7-8b56-f08a89dffe09-maas-api-openshift-ca" pod "9aac6c75-c3d7-4df7-8b56-f08a89dffe09" (UID: "9aac6c75-c3d7-4df7-8b56-f08a89dffe09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:24.832237 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:24.832218    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aac6c75-c3d7-4df7-8b56-f08a89dffe09-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:24.834768 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:24.834746    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aac6c75-c3d7-4df7-8b56-f08a89dffe09-maas-api-xks-ca" pod "9aac6c75-c3d7-4df7-8b56-f08a89dffe09" (UID: "9aac6c75-c3d7-4df7-8b56-f08a89dffe09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:24.834943 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:24.834919    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aac6c75-c3d7-4df7-8b56-f08a89dffe09-kube-api-access-6jhfr" pod "9aac6c75-c3d7-4df7-8b56-f08a89dffe09" (UID: "9aac6c75-c3d7-4df7-8b56-f08a89dffe09"). InnerVolumeSpecName "kube-api-access-6jhfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:24.934006 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:24.933944    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aac6c75-c3d7-4df7-8b56-f08a89dffe09-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:24.934006 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:24.933995    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jhfr\" (UniqueName: \"kubernetes.io/projected/9aac6c75-c3d7-4df7-8b56-f08a89dffe09-kube-api-access-6jhfr\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:25.086008 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:25.085973    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 20:06:25.086174 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:25.086031    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m" podUID="a977973c-341c-48ec-9df5-8035da148b90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 20:06:25.496919 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:25.496874    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 20:06:25.497087 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:25.496947    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb" podUID="e6fa742a-7bee-4a79-b5a1-aa33c24552f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 20:06:25.534266 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:25.534227    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1e9517-jmwhb"
Oct 30 20:06:25.534266 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:25.534243    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1e9517-jmwhb" event={"ID":"9aac6c75-c3d7-4df7-8b56-f08a89dffe09","Type":"ContainerDied","Data":"7ae04bae960d21e28d2af9e5ae4630fb7c6216b137770dbecaaa6e81248d4f55"}
Oct 30 20:06:25.534488 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:25.534294    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ae04bae960d21e28d2af9e5ae4630fb7c6216b137770dbecaaa6e81248d4f55"
Oct 30 20:06:26.090278 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:26.090241    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:26.090699 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:26.090655    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:26.091410 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:26.091389    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:06:26.764477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:26.762209    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:26.764477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:26.763658    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:26.764477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:26.764433    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:06:32.595014 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.594982    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"]
Oct 30 20:06:32.595658 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.595642    5958 cpu_manager.go:414] "containerMap: removing container" podUID="9aac6c75-c3d7-4df7-8b56-f08a89dffe09" containerName="revoke-keys"
Oct 30 20:06:32.595715 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.595659    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aac6c75-c3d7-4df7-8b56-f08a89dffe09" containerName="revoke-keys"
Oct 30 20:06:32.595751 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.595746    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aac6c75-c3d7-4df7-8b56-f08a89dffe09" containerName="revoke-keys"
Oct 30 20:06:32.599394 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.599371    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.647829 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.647803    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.647965 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.647851    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c65c\" (UniqueName: \"kubernetes.io/projected/cc641430-3d40-4d0d-b2c6-b551b090da1f-kube-api-access-2c65c\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.647965 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.647948    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.648092 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.648021    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.648152 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.648125    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.648204 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.648172    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.648310 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.648295    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.648375 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.648363    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.648429 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.648413    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc641430-3d40-4d0d-b2c6-b551b090da1f-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.705988 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.705958    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"]
Oct 30 20:06:32.749868 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.749833    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc641430-3d40-4d0d-b2c6-b551b090da1f-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.750014 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.749913    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.750014 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.749952    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c65c\" (UniqueName: \"kubernetes.io/projected/cc641430-3d40-4d0d-b2c6-b551b090da1f-kube-api-access-2c65c\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.750014 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.749985    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.750204 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.750026    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.750204 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.750057    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.750204 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.750095    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.750353 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.750218    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.750353 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.750280    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.750480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.750356    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-workload-certs\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.750711 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.750680    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc641430-3d40-4d0d-b2c6-b551b090da1f-istiod-ca-cert\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.750876 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.750757    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-credential-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.751064 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.750936    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-workload-socket\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.751064 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.751039    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-data\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.753025 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.753000    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-podinfo\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.753118 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.753095    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-envoy\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.812057 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.812022    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-token\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.812417 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.812391    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c65c\" (UniqueName: \"kubernetes.io/projected/cc641430-3d40-4d0d-b2c6-b551b090da1f-kube-api-access-2c65c\") pod \"e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") " pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:32.913473 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:32.913372    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:33.056808 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:33.056768    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e3a22a6-tf5nn"]
Oct 30 20:06:33.059720 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:33.059697    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e3a22a6-tf5nn"]
Oct 30 20:06:33.352045 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:33.352018    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"]
Oct 30 20:06:33.353533 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:06:33.353509    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc641430_3d40_4d0d_b2c6_b551b090da1f.slice/crio-c86b9da573ec1b2ad747f9163cb5fc892c85e7f74fc0694fc0c8a6d27d283354 WatchSource:0}: Error finding container c86b9da573ec1b2ad747f9163cb5fc892c85e7f74fc0694fc0c8a6d27d283354: Status 404 returned error can't find the container with id c86b9da573ec1b2ad747f9163cb5fc892c85e7f74fc0694fc0c8a6d27d283354
Oct 30 20:06:33.356146 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:33.356110    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:06:33.356227 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:33.356174    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:06:33.356227 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:33.356223    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:06:33.400838 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:33.400813    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2225fad5-f918-4bc2-9b8d-c35c84cf570f" path="/var/lib/kubelet/pods/2225fad5-f918-4bc2-9b8d-c35c84cf570f/volumes"
Oct 30 20:06:33.597210 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:33.597178    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89" event={"ID":"cc641430-3d40-4d0d-b2c6-b551b090da1f","Type":"ContainerStarted","Data":"94028ad1b4ccceb187ceb8c0143eece317dd6904d953ba6bcb74b45d5a6cfcb3"}
Oct 30 20:06:33.597601 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:33.597226    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89" event={"ID":"cc641430-3d40-4d0d-b2c6-b551b090da1f","Type":"ContainerStarted","Data":"c86b9da573ec1b2ad747f9163cb5fc892c85e7f74fc0694fc0c8a6d27d283354"}
Oct 30 20:06:33.721608 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:33.721499    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89" podStartSLOduration=1.721480157 podStartE2EDuration="1.721480157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:33.71872219 +0000 UTC m=+1432.450937581" watchObservedRunningTime="2026-10-30 20:06:33.721480157 +0000 UTC m=+1432.453695548"
Oct 30 20:06:33.913609 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:33.913570    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:34.915519 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:34.915444    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:06:34.915904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:34.915550    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89" podUID="cc641430-3d40-4d0d-b2c6-b551b090da1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 20:06:35.913904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:35.913864    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:06:35.914067 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:35.913933    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89" podUID="cc641430-3d40-4d0d-b2c6-b551b090da1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 20:06:36.606818 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:06:36.606778    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:06:36.913974 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:36.913888    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:06:36.913974 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:36.913945    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89" podUID="cc641430-3d40-4d0d-b2c6-b551b090da1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 20:06:37.914521 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:37.914485    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:06:37.914995 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:37.914552    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89" podUID="cc641430-3d40-4d0d-b2c6-b551b090da1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 20:06:38.914424 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:38.914388    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:06:38.914643 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:38.914477    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89" podUID="cc641430-3d40-4d0d-b2c6-b551b090da1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 20:06:39.914568 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:39.914531    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 30 20:06:39.914757 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:39.914592    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89" podUID="cc641430-3d40-4d0d-b2c6-b551b090da1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 30 20:06:40.918267 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:40.918234    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:40.918751 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:40.918732    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:40.919415 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:40.919397    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:06:46.407980 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.407948    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"]
Oct 30 20:06:46.413114 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.413094    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.458036 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.457998    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"]
Oct 30 20:06:46.460181 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.460153    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.460280 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.460192    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.460336 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.460312    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.460372 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.460348    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.460425 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.460412    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7fc5\" (UniqueName: \"kubernetes.io/projected/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-kube-api-access-v7fc5\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.460604 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.460590    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.460646 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.460622    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.460697 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.460648    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.460754 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.460734    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.462185 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.462167    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.509120 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.509092    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"]
Oct 30 20:06:46.562940 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.562899    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.563098 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.562968    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.563098 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.563011    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.563098 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.563036    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.563098 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.563054    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.563098 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.563076    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.563098 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.563096    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.563360 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.563123    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df7304e3-998f-4610-acf5-aee21d0ee2ad-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.563360 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.563222    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.563360 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.563278    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44hrn\" (UniqueName: \"kubernetes.io/projected/df7304e3-998f-4610-acf5-aee21d0ee2ad-kube-api-access-44hrn\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.563543 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.563373    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.563543 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.563411    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.563543 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.563527    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.563686 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.563552    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.563686 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.563603    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.563686 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.563633    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.563686 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.563676    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7fc5\" (UniqueName: \"kubernetes.io/projected/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-kube-api-access-v7fc5\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.563944 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.563724    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.564410 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.564388    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.564779 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.564759    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.564875 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.564850    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.565047 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.565024    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.565128 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.565108    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.566252 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.566230    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"]
Oct 30 20:06:46.567955 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.567929    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.568327 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.568305    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.639142 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.639117    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.639666 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.639646    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7fc5\" (UniqueName: \"kubernetes.io/projected/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-kube-api-access-v7fc5\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-k96dk\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.665243 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.665148    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.665243 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.665206    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.665473 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.665288    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.665473 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.665342    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.665473 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.665418    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.665645 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.665443    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.665645 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.665510    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df7304e3-998f-4610-acf5-aee21d0ee2ad-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.665645 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.665603    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.665804 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.665661    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44hrn\" (UniqueName: \"kubernetes.io/projected/df7304e3-998f-4610-acf5-aee21d0ee2ad-kube-api-access-44hrn\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.665804 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.665775    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-credential-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.665804 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.665799    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-workload-socket\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.665951 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.665885    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-workload-certs\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.666054 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.666036    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-data\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.666433 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.666405    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df7304e3-998f-4610-acf5-aee21d0ee2ad-istiod-ca-cert\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.667964 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.667936    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-envoy\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.668622 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.668600    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-podinfo\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.726668 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.726640    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:46.727326 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.727296    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-token\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.727414 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.727380    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44hrn\" (UniqueName: \"kubernetes.io/projected/df7304e3-998f-4610-acf5-aee21d0ee2ad-kube-api-access-44hrn\") pod \"e2e-worker-w6-f75272-openshift-default-649458586c-xq89x\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") " pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:46.775501 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:46.775478    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:47.172632 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:47.172600    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"]
Oct 30 20:06:47.206682 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:06:47.206656    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c2eca7f_c86c_427e_b1c5_1d33ded45e42.slice/crio-c9af10c6606634bb14ecb1b44a31104b05837d9fc1888e0947cd3c3c6fe6b500 WatchSource:0}: Error finding container c9af10c6606634bb14ecb1b44a31104b05837d9fc1888e0947cd3c3c6fe6b500: Status 404 returned error can't find the container with id c9af10c6606634bb14ecb1b44a31104b05837d9fc1888e0947cd3c3c6fe6b500
Oct 30 20:06:47.209210 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:47.209174    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:06:47.209309 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:47.209256    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:06:47.209309 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:47.209295    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:06:47.225962 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:47.225937    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"]
Oct 30 20:06:47.226517 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:06:47.226483    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7304e3_998f_4610_acf5_aee21d0ee2ad.slice/crio-8f873a8f39deb6fdddb74a2a283a0f713b863bc195426f31fae6bbd7cf9e7d5c WatchSource:0}: Error finding container 8f873a8f39deb6fdddb74a2a283a0f713b863bc195426f31fae6bbd7cf9e7d5c: Status 404 returned error can't find the container with id 8f873a8f39deb6fdddb74a2a283a0f713b863bc195426f31fae6bbd7cf9e7d5c
Oct 30 20:06:47.228884 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:47.228859    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:06:47.228934 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:47.228921    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:06:47.228968 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:47.228953    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:06:47.728602 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:47.728565    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk" event={"ID":"7c2eca7f-c86c-427e-b1c5-1d33ded45e42","Type":"ContainerStarted","Data":"40fc7497374a4d34d0d94b9b1d14f7f5cff7d1e0e7f7404088d46fb1f3899744"}
Oct 30 20:06:47.729052 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:47.728646    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk" event={"ID":"7c2eca7f-c86c-427e-b1c5-1d33ded45e42","Type":"ContainerStarted","Data":"c9af10c6606634bb14ecb1b44a31104b05837d9fc1888e0947cd3c3c6fe6b500"}
Oct 30 20:06:47.730301 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:47.730270    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x" event={"ID":"df7304e3-998f-4610-acf5-aee21d0ee2ad","Type":"ContainerStarted","Data":"5d8d7c0292f29948cbce12a9ae37ee3d98de51b756739d9062d09c47afef39a8"}
Oct 30 20:06:47.730445 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:47.730326    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x" event={"ID":"df7304e3-998f-4610-acf5-aee21d0ee2ad","Type":"ContainerStarted","Data":"8f873a8f39deb6fdddb74a2a283a0f713b863bc195426f31fae6bbd7cf9e7d5c"}
Oct 30 20:06:47.776114 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:47.776082    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:47.863475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:47.859910    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk" podStartSLOduration=1.859894878 podStartE2EDuration="1.859894878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:47.853709534 +0000 UTC m=+1446.585924938" watchObservedRunningTime="2026-10-30 20:06:47.859894878 +0000 UTC m=+1446.592110342"
Oct 30 20:06:48.732135 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:48.731160    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:48.732783 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:48.732747    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:06:48.732963 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:48.732942    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk" podUID="7c2eca7f-c86c-427e-b1c5-1d33ded45e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 20:06:48.795077 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:48.795044    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 20:06:48.795376 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:48.795333    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x" podUID="df7304e3-998f-4610-acf5-aee21d0ee2ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 20:06:49.731533 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:49.730576    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:06:49.731533 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:49.730636    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk" podUID="7c2eca7f-c86c-427e-b1c5-1d33ded45e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 20:06:49.781067 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:49.780943    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 20:06:49.781067 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:49.781003    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x" podUID="df7304e3-998f-4610-acf5-aee21d0ee2ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 20:06:50.176478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:50.173607    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:50.625477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:50.624774    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:50.728126 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:50.728088    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:06:50.728401 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:50.728373    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk" podUID="7c2eca7f-c86c-427e-b1c5-1d33ded45e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 20:06:50.766486 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:50.762947    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" podUID="6e20c97b-66ce-485b-ba8d-90c2d9e1fa76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:50.780480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:50.776477    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 20:06:50.780480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:50.776554    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x" podUID="df7304e3-998f-4610-acf5-aee21d0ee2ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 20:06:50.780480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:50.779528    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:50.837479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:50.836140    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 30 20:06:50.988530 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:50.985560    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:51.024476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:51.019575    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:51.053625 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:51.053568    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 20:06:51.175291 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:51.175235    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:51.175842 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:51.175736    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:06:51.205638 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:51.205595    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 20:06:51.304354 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:51.304212    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x" podStartSLOduration=5.304175765 podStartE2EDuration="5.304175765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:47.988852295 +0000 UTC m=+1446.721067685" watchObservedRunningTime="2026-10-30 20:06:51.304175765 +0000 UTC m=+1450.036391155"
Oct 30 20:06:51.332879 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:51.332535    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:51.473475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:51.469862    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 30 20:06:51.730760 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:51.730724    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:06:51.731045 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:51.731017    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk" podUID="7c2eca7f-c86c-427e-b1c5-1d33ded45e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 20:06:51.783737 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:51.783694    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:51.784271 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:51.783853    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 20:06:51.784954 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:51.784910    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x" podUID="df7304e3-998f-4610-acf5-aee21d0ee2ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 20:06:51.785134 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:51.784541    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:06:51.943092 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:51.943044    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:51.971928 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:51.971898    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:06:51.997071 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:51.996972    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:52.010123 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:51.998701    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 20:06:52.023979 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:52.023938    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:52.024607 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:52.024583    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:06:52.095618 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:52.095571    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 20:06:52.215478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:52.195168    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 20:06:52.219167 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:52.208536    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:52.220382 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:52.220361    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 20:06:52.278814 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:52.278707    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:52.351476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:52.347714    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:52.382479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:52.368658    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:06:52.495112 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:52.493901    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:52.510480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:52.495632    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:06:52.510480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:52.496787    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:06:52.581602 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:52.581558    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:06:52.585475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:52.581877    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 20:06:52.718484 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:52.714145    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:06:52.718484 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:52.714210    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:06:52.718484 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:52.717132    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 20:06:52.725478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:52.719077    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 20:06:52.734499 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:52.733640    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:06:52.734499 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:52.733704    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk" podUID="7c2eca7f-c86c-427e-b1c5-1d33ded45e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 20:06:52.778784 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:52.778751    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:06:52.779670 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:52.779649    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:06:52.803611 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:52.803586    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 20:06:52.803821 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:52.803801    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x" podUID="df7304e3-998f-4610-acf5-aee21d0ee2ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 20:06:53.727600 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:53.727565    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:06:53.727985 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:53.727611    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk" podUID="7c2eca7f-c86c-427e-b1c5-1d33ded45e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 20:06:53.799025 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:53.798977    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 20:06:53.799215 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:53.799049    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x" podUID="df7304e3-998f-4610-acf5-aee21d0ee2ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 20:06:54.727634 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:54.727596    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 20:06:54.728130 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:54.727653    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk" podUID="7c2eca7f-c86c-427e-b1c5-1d33ded45e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 20:06:54.776213 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:54.776174    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 20:06:54.776389 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:54.776226    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x" podUID="df7304e3-998f-4610-acf5-aee21d0ee2ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 20:06:55.572857 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:55.572820    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55e619f8-27d7k"]
Oct 30 20:06:55.579194 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:55.579170    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e619f8-27d7k"
Oct 30 20:06:55.635367 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:55.635332    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55e619f8-27d7k"]
Oct 30 20:06:55.679977 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:55.679951    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:06:55.680186 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:55.679961    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:06:55.683966 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:55.683941    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ada3b129-dc27-4d16-8707-e28d00d1cc2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55e619f8-27d7k\" (UID: \"ada3b129-dc27-4d16-8707-e28d00d1cc2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e619f8-27d7k"
Oct 30 20:06:55.684088 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:55.684040    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ada3b129-dc27-4d16-8707-e28d00d1cc2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55e619f8-27d7k\" (UID: \"ada3b129-dc27-4d16-8707-e28d00d1cc2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e619f8-27d7k"
Oct 30 20:06:55.684264 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:55.684246    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmbb7\" (UniqueName: \"kubernetes.io/projected/ada3b129-dc27-4d16-8707-e28d00d1cc2d-kube-api-access-pmbb7\") pod \"maas-api-revoke-sub-55e619f8-27d7k\" (UID: \"ada3b129-dc27-4d16-8707-e28d00d1cc2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e619f8-27d7k"
Oct 30 20:06:55.785854 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:55.785810    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmbb7\" (UniqueName: \"kubernetes.io/projected/ada3b129-dc27-4d16-8707-e28d00d1cc2d-kube-api-access-pmbb7\") pod \"maas-api-revoke-sub-55e619f8-27d7k\" (UID: \"ada3b129-dc27-4d16-8707-e28d00d1cc2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e619f8-27d7k"
Oct 30 20:06:55.786289 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:55.785992    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ada3b129-dc27-4d16-8707-e28d00d1cc2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55e619f8-27d7k\" (UID: \"ada3b129-dc27-4d16-8707-e28d00d1cc2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e619f8-27d7k"
Oct 30 20:06:55.786289 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:55.786084    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ada3b129-dc27-4d16-8707-e28d00d1cc2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55e619f8-27d7k\" (UID: \"ada3b129-dc27-4d16-8707-e28d00d1cc2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e619f8-27d7k"
Oct 30 20:06:55.787204 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:55.787174    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ada3b129-dc27-4d16-8707-e28d00d1cc2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55e619f8-27d7k\" (UID: \"ada3b129-dc27-4d16-8707-e28d00d1cc2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e619f8-27d7k"
Oct 30 20:06:55.792493 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:55.789077    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ada3b129-dc27-4d16-8707-e28d00d1cc2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55e619f8-27d7k\" (UID: \"ada3b129-dc27-4d16-8707-e28d00d1cc2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e619f8-27d7k"
Oct 30 20:06:55.848728 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:55.848672    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmbb7\" (UniqueName: \"kubernetes.io/projected/ada3b129-dc27-4d16-8707-e28d00d1cc2d-kube-api-access-pmbb7\") pod \"maas-api-revoke-sub-55e619f8-27d7k\" (UID: \"ada3b129-dc27-4d16-8707-e28d00d1cc2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e619f8-27d7k"
Oct 30 20:06:55.894105 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:55.894079    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e619f8-27d7k"
Oct 30 20:06:56.293831 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:06:56.290864    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada3b129_dc27_4d16_8707_e28d00d1cc2d.slice/crio-285bba32ee2bd9fc99a677dc6baf86101cb41a687d06fc36a746dcb2c15bf2ee WatchSource:0}: Error finding container 285bba32ee2bd9fc99a677dc6baf86101cb41a687d06fc36a746dcb2c15bf2ee: Status 404 returned error can't find the container with id 285bba32ee2bd9fc99a677dc6baf86101cb41a687d06fc36a746dcb2c15bf2ee
Oct 30 20:06:56.295921 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:56.295851    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55e619f8-27d7k"]
Oct 30 20:06:56.332394 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:56.332368    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e619f8-27d7k" event={"ID":"ada3b129-dc27-4d16-8707-e28d00d1cc2d","Type":"ContainerStarted","Data":"285bba32ee2bd9fc99a677dc6baf86101cb41a687d06fc36a746dcb2c15bf2ee"}
Oct 30 20:06:56.772910 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:56.770529    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:06:56.772910 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:56.770583    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk" podUID="7c2eca7f-c86c-427e-b1c5-1d33ded45e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:56.821987 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:56.821961    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:06:56.822383 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:56.822359    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x" podUID="df7304e3-998f-4610-acf5-aee21d0ee2ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:06:57.059344 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:57.057132    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:57.059344 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:57.058588    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:57.060237 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:57.060201    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:06:57.262185 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:57.262157    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:57.262476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:57.262445    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:57.263254 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:57.263239    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:06:57.340894 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:57.340828    5958 generic.go:356] "Generic (PLEG): container finished" podID="ada3b129-dc27-4d16-8707-e28d00d1cc2d" containerID="a12a5d7582cd0b396392178d0f264c7e97b5149b2b7007ad6d77275bab3d6ea2" exitCode=0
Oct 30 20:06:57.341039 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:57.341001    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e619f8-27d7k" event={"ID":"ada3b129-dc27-4d16-8707-e28d00d1cc2d","Type":"ContainerDied","Data":"a12a5d7582cd0b396392178d0f264c7e97b5149b2b7007ad6d77275bab3d6ea2"}
Oct 30 20:06:58.512756 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:58.512722    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e619f8-27d7k"
Oct 30 20:06:58.640578 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:58.640501    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ada3b129-dc27-4d16-8707-e28d00d1cc2d-kube-api-access-pmbb7\" (UniqueName: \"kubernetes.io/projected/ada3b129-dc27-4d16-8707-e28d00d1cc2d-kube-api-access-pmbb7\") pod \"ada3b129-dc27-4d16-8707-e28d00d1cc2d\" (UID: \"ada3b129-dc27-4d16-8707-e28d00d1cc2d\") "
Oct 30 20:06:58.640868 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:58.640840    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ada3b129-dc27-4d16-8707-e28d00d1cc2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ada3b129-dc27-4d16-8707-e28d00d1cc2d-maas-api-openshift-ca\") pod \"ada3b129-dc27-4d16-8707-e28d00d1cc2d\" (UID: \"ada3b129-dc27-4d16-8707-e28d00d1cc2d\") "
Oct 30 20:06:58.641089 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:58.641064    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ada3b129-dc27-4d16-8707-e28d00d1cc2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ada3b129-dc27-4d16-8707-e28d00d1cc2d-maas-api-xks-ca\") pod \"ada3b129-dc27-4d16-8707-e28d00d1cc2d\" (UID: \"ada3b129-dc27-4d16-8707-e28d00d1cc2d\") "
Oct 30 20:06:58.641392 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:58.641362    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ada3b129-dc27-4d16-8707-e28d00d1cc2d-maas-api-openshift-ca" pod "ada3b129-dc27-4d16-8707-e28d00d1cc2d" (UID: "ada3b129-dc27-4d16-8707-e28d00d1cc2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:58.641837 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:58.641812    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ada3b129-dc27-4d16-8707-e28d00d1cc2d-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:58.643657 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:58.643635    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ada3b129-dc27-4d16-8707-e28d00d1cc2d-maas-api-xks-ca" pod "ada3b129-dc27-4d16-8707-e28d00d1cc2d" (UID: "ada3b129-dc27-4d16-8707-e28d00d1cc2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:58.644131 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:58.644110    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada3b129-dc27-4d16-8707-e28d00d1cc2d-kube-api-access-pmbb7" pod "ada3b129-dc27-4d16-8707-e28d00d1cc2d" (UID: "ada3b129-dc27-4d16-8707-e28d00d1cc2d"). InnerVolumeSpecName "kube-api-access-pmbb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:58.743227 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:58.743198    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ada3b129-dc27-4d16-8707-e28d00d1cc2d-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:58.743227 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:58.743224    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmbb7\" (UniqueName: \"kubernetes.io/projected/ada3b129-dc27-4d16-8707-e28d00d1cc2d-kube-api-access-pmbb7\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:59.355999 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:59.355970    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e619f8-27d7k"
Oct 30 20:06:59.356168 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:59.356005    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55e619f8-27d7k" event={"ID":"ada3b129-dc27-4d16-8707-e28d00d1cc2d","Type":"ContainerDied","Data":"285bba32ee2bd9fc99a677dc6baf86101cb41a687d06fc36a746dcb2c15bf2ee"}
Oct 30 20:06:59.356168 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:06:59.356075    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="285bba32ee2bd9fc99a677dc6baf86101cb41a687d06fc36a746dcb2c15bf2ee"
Oct 30 20:07:03.098685 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.098648    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"]
Oct 30 20:07:03.099318 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.099302    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ada3b129-dc27-4d16-8707-e28d00d1cc2d" containerName="revoke-keys"
Oct 30 20:07:03.099318 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.099318    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ada3b129-dc27-4d16-8707-e28d00d1cc2d" containerName="revoke-keys"
Oct 30 20:07:03.099444 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.099435    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ada3b129-dc27-4d16-8707-e28d00d1cc2d" containerName="revoke-keys"
Oct 30 20:07:03.103015 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.102995    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.163837 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.163803    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"]
Oct 30 20:07:03.197075 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.197039    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl82s\" (UniqueName: \"kubernetes.io/projected/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-kube-api-access-hl82s\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.197289 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.197257    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.197368 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.197312    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.197368 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.197349    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.197506 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.197407    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.197506 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.197498    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.197595 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.197550    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.197595 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.197578    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.197695 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.197664    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.299075 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.299041    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.299227 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.299096    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.299227 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.299147    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.299227 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.299169    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.299416 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.299238    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.299416 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.299285    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl82s\" (UniqueName: \"kubernetes.io/projected/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-kube-api-access-hl82s\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.299576 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.299491    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.299576 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.299541    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.299576 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.299553    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.299727 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.299576    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.299727 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.299578    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.299829 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.299731    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.299887 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.299848    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.300089 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.300057    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.302092 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.302067    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.302515 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.302498    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.361180 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.361130    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.361405 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.361388    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl82s\" (UniqueName: \"kubernetes.io/projected/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-kube-api-access-hl82s\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.421007 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.420980    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:03.853088 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.853059    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"]
Oct 30 20:07:03.853589 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:07:03.853570    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a44ce4e_0ecf_4b57_b400_abef5d8f2dd2.slice/crio-f2d54e570beec1a411a06c10d43bf5ea5b2dafe3a584ab4d851900f0de0d51f2 WatchSource:0}: Error finding container f2d54e570beec1a411a06c10d43bf5ea5b2dafe3a584ab4d851900f0de0d51f2: Status 404 returned error can't find the container with id f2d54e570beec1a411a06c10d43bf5ea5b2dafe3a584ab4d851900f0de0d51f2
Oct 30 20:07:03.855883 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.855848    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:07:03.855978 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.855914    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:07:03.855978 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:03.855943    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:07:04.390416 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:04.390374    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g" event={"ID":"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2","Type":"ContainerStarted","Data":"35b0a1d334ce37916123a56d716f7f16a0d2e915c615021be2f2451bf43cd9e9"}
Oct 30 20:07:04.390878 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:04.390482    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g" event={"ID":"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2","Type":"ContainerStarted","Data":"f2d54e570beec1a411a06c10d43bf5ea5b2dafe3a584ab4d851900f0de0d51f2"}
Oct 30 20:07:04.421649 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:04.421628    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:04.422809 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:04.422781    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:07:04.422937 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:04.422823    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g" podUID="0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 20:07:04.529875 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:04.529820    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g" podStartSLOduration=1.52980383 podStartE2EDuration="1.52980383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:04.515336032 +0000 UTC m=+1463.247551422" watchObservedRunningTime="2026-10-30 20:07:04.52980383 +0000 UTC m=+1463.262019219"
Oct 30 20:07:05.422058 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:05.422028    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:07:05.422487 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:05.422085    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g" podUID="0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 20:07:06.421669 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:06.421638    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:07:06.421838 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:06.421694    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g" podUID="0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 20:07:06.609185 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:07:06.609152    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:07:07.421499 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:07.421436    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:07:07.421689 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:07.421519    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g" podUID="0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 20:07:08.421443 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:08.421407    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:07:08.421848 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:08.421501    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g" podUID="0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 20:07:09.422111 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:09.422071    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:07:09.422616 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:09.422142    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g" podUID="0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 20:07:10.421902 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:10.421871    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 30 20:07:10.422071 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:10.421918    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g" podUID="0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 30 20:07:11.425516 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:11.425485    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:11.425888 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:11.425862    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:11.426467 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:11.426434    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:07:18.240593 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.240538    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"]
Oct 30 20:07:18.245550 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.245527    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.351812 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.351772    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"]
Oct 30 20:07:18.431469 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.423569    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.431469 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.423641    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86hxt\" (UniqueName: \"kubernetes.io/projected/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-kube-api-access-86hxt\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.431469 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.423684    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.431469 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.423745    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.431469 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.423775    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.431469 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.423813    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.431469 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.423851    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.431469 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.423980    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.431469 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.424056    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.525631 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.525551    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.525792 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.525650    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86hxt\" (UniqueName: \"kubernetes.io/projected/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-kube-api-access-86hxt\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.525792 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.525732    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.525925 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.525788    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.525925 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.525832    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.525925 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.525866    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.526083 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.526039    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.526278 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.526238    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.526632 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.526582    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.526905 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.526879    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.527038 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.526556    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.527038 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.526991    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.527172 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.526614    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.527362 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.527340    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.528554 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.528531    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.529084 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.529059    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.587360 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.587340    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86hxt\" (UniqueName: \"kubernetes.io/projected/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-kube-api-access-86hxt\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.588174 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.588153    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:18.860457 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:18.860430    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:19.307815 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:19.307762    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"]
Oct 30 20:07:19.308149 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:07:19.307878    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b2fbafd_bdd0_48b0_bdd9_73b84d6951f9.slice/crio-61edda50818f495bee3372183ecdb8f8096d2dbfaf7725c8c8926ce31c393a44 WatchSource:0}: Error finding container 61edda50818f495bee3372183ecdb8f8096d2dbfaf7725c8c8926ce31c393a44: Status 404 returned error can't find the container with id 61edda50818f495bee3372183ecdb8f8096d2dbfaf7725c8c8926ce31c393a44
Oct 30 20:07:19.310920 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:19.310880    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:07:19.311034 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:19.310959    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:07:19.311034 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:19.311005    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:07:19.497618 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:19.497577    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5" event={"ID":"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9","Type":"ContainerStarted","Data":"c90fd6a988ed86fdd24c7455be1733b2418ab0d8aa3aaa0b9f94ba8b58b3f36f"}
Oct 30 20:07:19.497777 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:19.497635    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5" event={"ID":"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9","Type":"ContainerStarted","Data":"61edda50818f495bee3372183ecdb8f8096d2dbfaf7725c8c8926ce31c393a44"}
Oct 30 20:07:19.627399 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:19.627352    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5" podStartSLOduration=1.627339842 podStartE2EDuration="1.627339842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:19.622505713 +0000 UTC m=+1478.354721104" watchObservedRunningTime="2026-10-30 20:07:19.627339842 +0000 UTC m=+1478.359555231"
Oct 30 20:07:19.861143 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:19.861102    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:20.869702 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:20.869667    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 20:07:20.870316 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:20.870282    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5" podUID="0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 20:07:21.794581 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:21.794537    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:21.863071 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:21.863024    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 20:07:21.863399 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:21.863347    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5" podUID="0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 20:07:22.284477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:22.281551    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:22.503365 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:22.502746    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Oct 30 20:07:22.576004 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:22.575940    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:22.579528 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:22.577738    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:22.681875 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:22.679853    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded"
Oct 30 20:07:22.701892 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:22.701098    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:22.756704 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:22.756028    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:22.756704 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:22.756391    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:22.766850 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:22.766535    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" podUID="6e20c97b-66ce-485b-ba8d-90c2d9e1fa76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded"
Oct 30 20:07:22.811478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:22.799027    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:22.811478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:22.799477    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 20:07:22.811478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:22.799814    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:07:22.831371 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:22.829984    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:22.889375 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:22.888845    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 20:07:22.889375 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:22.888903    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5" podUID="0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 20:07:22.935976 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:22.930815    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:07:22.935976 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:22.930881    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf" podUID="48c11b72-fdc6-4d2c-849b-136a5a53259d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:23.369585 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:23.287541    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:23.369585 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:23.288000    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:07:23.590477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:23.579362    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:23.597533 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:23.594818    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 20:07:23.597533 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:23.596883    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Oct 30 20:07:23.721834 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:23.716094    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:23.780514 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:23.774191    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:23.780514 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:23.774598    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 20:07:23.780514 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:23.779534    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" podUID="6e20c97b-66ce-485b-ba8d-90c2d9e1fa76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:23.841034 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:23.840998    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 30 20:07:23.841627 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:23.841597    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:23.867917 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:23.867880    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 20:07:23.873750 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:23.873704    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5" podUID="0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 20:07:24.091798 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:24.091765    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:07:24.121760 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:24.121732    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 20:07:24.123017 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:24.122997    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:07:24.157482 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:24.152604    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:07:24.177637 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:24.177609    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 20:07:24.177909 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:24.177891    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 20:07:24.181479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:24.178038    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 20:07:24.200575 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:24.200540    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:07:24.330839 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:24.330791    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:24.666578 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:24.666538    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 20:07:24.680237 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:24.677862    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 20:07:24.680237 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:24.678509    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Oct 30 20:07:24.719647 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:24.715101    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:07:24.719647 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:24.715159    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675" podUID="8bfda83c-7138-4974-ae1c-67fa552ac430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:24.719892 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:24.719684    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:24.796174 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:24.796119    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:24.796364 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:24.796231    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" podUID="6e20c97b-66ce-485b-ba8d-90c2d9e1fa76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded"
Oct 30 20:07:24.798327 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:24.797380    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:24.799481 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:24.798541    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 20:07:24.880490 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:24.873956    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 20:07:24.880490 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:24.874017    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5" podUID="0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 20:07:24.921607 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:24.917761    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:24.937736 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:24.937697    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 30 20:07:24.962662 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:24.962617    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:25.052864 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:25.052824    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:07:25.112443 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:25.112413    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 20:07:25.308674 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:25.307930    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:07:25.315315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:25.315275    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:07:25.350477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:25.327961    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:07:25.356346 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:25.356319    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 20:07:25.362036 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:25.362014    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:07:25.387117 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:25.385656    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:07:25.388383 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:25.388357    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 20:07:25.389867 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:25.389769    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 20:07:25.865478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:25.864935    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 20:07:25.865478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:25.864993    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5" podUID="0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 20:07:26.063467 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:26.063404    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e21eac26-fpwt9"]
Oct 30 20:07:26.068204 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:26.068170    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e21eac26-fpwt9"]
Oct 30 20:07:26.863122 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:26.863084    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 20:07:26.863407 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:26.863378    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5" podUID="0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 20:07:27.400144 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:27.400107    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="000a5b9e-d346-4782-ac64-e31a79f2e9ae" path="/var/lib/kubelet/pods/000a5b9e-d346-4782-ac64-e31a79f2e9ae/volumes"
Oct 30 20:07:27.860790 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:27.860752    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 20:07:27.860988 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:27.860802    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5" podUID="0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 20:07:29.892751 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:29.892710    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:07:29.893252 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:29.892785    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5" podUID="0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:29.905336 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:29.905308    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:29.905724 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:29.905700    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:29.907737 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:29.907711    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:07:30.859944 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:30.859908    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058eadc3-tp8m7"]
Oct 30 20:07:30.864257 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:30.864239    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058eadc3-tp8m7"
Oct 30 20:07:30.924040 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:30.924006    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058eadc3-tp8m7"]
Oct 30 20:07:30.968312 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:30.968275    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:07:30.968504 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:30.968328    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:07:31.033033 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:31.033001    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c786d26b-042b-42eb-9520-434658f959d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-058eadc3-tp8m7\" (UID: \"c786d26b-042b-42eb-9520-434658f959d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058eadc3-tp8m7"
Oct 30 20:07:31.033208 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:31.033133    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlmss\" (UniqueName: \"kubernetes.io/projected/c786d26b-042b-42eb-9520-434658f959d1-kube-api-access-rlmss\") pod \"maas-api-revoke-sub-058eadc3-tp8m7\" (UID: \"c786d26b-042b-42eb-9520-434658f959d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058eadc3-tp8m7"
Oct 30 20:07:31.033378 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:31.033356    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c786d26b-042b-42eb-9520-434658f959d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-058eadc3-tp8m7\" (UID: \"c786d26b-042b-42eb-9520-434658f959d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058eadc3-tp8m7"
Oct 30 20:07:31.134575 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:31.134501    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c786d26b-042b-42eb-9520-434658f959d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-058eadc3-tp8m7\" (UID: \"c786d26b-042b-42eb-9520-434658f959d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058eadc3-tp8m7"
Oct 30 20:07:31.134733 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:31.134613    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c786d26b-042b-42eb-9520-434658f959d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-058eadc3-tp8m7\" (UID: \"c786d26b-042b-42eb-9520-434658f959d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058eadc3-tp8m7"
Oct 30 20:07:31.134856 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:31.134819    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlmss\" (UniqueName: \"kubernetes.io/projected/c786d26b-042b-42eb-9520-434658f959d1-kube-api-access-rlmss\") pod \"maas-api-revoke-sub-058eadc3-tp8m7\" (UID: \"c786d26b-042b-42eb-9520-434658f959d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058eadc3-tp8m7"
Oct 30 20:07:31.135331 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:31.135276    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c786d26b-042b-42eb-9520-434658f959d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-058eadc3-tp8m7\" (UID: \"c786d26b-042b-42eb-9520-434658f959d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058eadc3-tp8m7"
Oct 30 20:07:31.137014 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:31.136993    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c786d26b-042b-42eb-9520-434658f959d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-058eadc3-tp8m7\" (UID: \"c786d26b-042b-42eb-9520-434658f959d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058eadc3-tp8m7"
Oct 30 20:07:31.197187 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:31.197157    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlmss\" (UniqueName: \"kubernetes.io/projected/c786d26b-042b-42eb-9520-434658f959d1-kube-api-access-rlmss\") pod \"maas-api-revoke-sub-058eadc3-tp8m7\" (UID: \"c786d26b-042b-42eb-9520-434658f959d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058eadc3-tp8m7"
Oct 30 20:07:31.479039 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:31.478947    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058eadc3-tp8m7"
Oct 30 20:07:31.871833 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:31.871800    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058eadc3-tp8m7"]
Oct 30 20:07:31.873395 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:07:31.873367    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc786d26b_042b_42eb_9520_434658f959d1.slice/crio-f85622456cb121fa6706ac6827456f20bd98d92ae7bfe05c4584b0ab2c31e059 WatchSource:0}: Error finding container f85622456cb121fa6706ac6827456f20bd98d92ae7bfe05c4584b0ab2c31e059: Status 404 returned error can't find the container with id f85622456cb121fa6706ac6827456f20bd98d92ae7bfe05c4584b0ab2c31e059
Oct 30 20:07:32.046120 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:32.046066    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058eadc3-tp8m7" event={"ID":"c786d26b-042b-42eb-9520-434658f959d1","Type":"ContainerStarted","Data":"f85622456cb121fa6706ac6827456f20bd98d92ae7bfe05c4584b0ab2c31e059"}
Oct 30 20:07:33.054999 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:33.054962    5958 generic.go:356] "Generic (PLEG): container finished" podID="c786d26b-042b-42eb-9520-434658f959d1" containerID="25d540e068026222690a8cb912c0d86c2e31ca83ea1340c12060c89e1f78348b" exitCode=0
Oct 30 20:07:33.055403 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:33.055041    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058eadc3-tp8m7" event={"ID":"c786d26b-042b-42eb-9520-434658f959d1","Type":"ContainerDied","Data":"25d540e068026222690a8cb912c0d86c2e31ca83ea1340c12060c89e1f78348b"}
Oct 30 20:07:34.259112 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:34.259082    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058eadc3-tp8m7"
Oct 30 20:07:34.384261 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:34.384219    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c786d26b-042b-42eb-9520-434658f959d1-kube-api-access-rlmss\" (UniqueName: \"kubernetes.io/projected/c786d26b-042b-42eb-9520-434658f959d1-kube-api-access-rlmss\") pod \"c786d26b-042b-42eb-9520-434658f959d1\" (UID: \"c786d26b-042b-42eb-9520-434658f959d1\") "
Oct 30 20:07:34.384548 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:34.384339    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c786d26b-042b-42eb-9520-434658f959d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c786d26b-042b-42eb-9520-434658f959d1-maas-api-openshift-ca\") pod \"c786d26b-042b-42eb-9520-434658f959d1\" (UID: \"c786d26b-042b-42eb-9520-434658f959d1\") "
Oct 30 20:07:34.384802 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:34.384777    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c786d26b-042b-42eb-9520-434658f959d1-maas-api-openshift-ca" pod "c786d26b-042b-42eb-9520-434658f959d1" (UID: "c786d26b-042b-42eb-9520-434658f959d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:07:34.384949 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:34.384932    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c786d26b-042b-42eb-9520-434658f959d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c786d26b-042b-42eb-9520-434658f959d1-maas-api-xks-ca\") pod \"c786d26b-042b-42eb-9520-434658f959d1\" (UID: \"c786d26b-042b-42eb-9520-434658f959d1\") "
Oct 30 20:07:34.385581 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:34.385554    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c786d26b-042b-42eb-9520-434658f959d1-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:34.389653 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:34.389599    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c786d26b-042b-42eb-9520-434658f959d1-kube-api-access-rlmss" pod "c786d26b-042b-42eb-9520-434658f959d1" (UID: "c786d26b-042b-42eb-9520-434658f959d1"). InnerVolumeSpecName "kube-api-access-rlmss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:07:34.404606 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:34.400547    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c786d26b-042b-42eb-9520-434658f959d1-maas-api-xks-ca" pod "c786d26b-042b-42eb-9520-434658f959d1" (UID: "c786d26b-042b-42eb-9520-434658f959d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:07:34.487418 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:34.487381    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c786d26b-042b-42eb-9520-434658f959d1-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:34.487418 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:34.487419    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlmss\" (UniqueName: \"kubernetes.io/projected/c786d26b-042b-42eb-9520-434658f959d1-kube-api-access-rlmss\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:35.071314 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:35.071284    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058eadc3-tp8m7"
Oct 30 20:07:35.071314 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:35.071305    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058eadc3-tp8m7" event={"ID":"c786d26b-042b-42eb-9520-434658f959d1","Type":"ContainerDied","Data":"f85622456cb121fa6706ac6827456f20bd98d92ae7bfe05c4584b0ab2c31e059"}
Oct 30 20:07:35.071555 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:35.071354    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f85622456cb121fa6706ac6827456f20bd98d92ae7bfe05c4584b0ab2c31e059"
Oct 30 20:07:36.076943 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:36.076905    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3227a6e-mprf8"]
Oct 30 20:07:36.126655 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:36.126619    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3227a6e-mprf8"]
Oct 30 20:07:36.606694 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:07:36.606655    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:07:37.400541 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:37.400512    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0609b754-971f-4749-87ca-dedea0b03b90" path="/var/lib/kubelet/pods/0609b754-971f-4749-87ca-dedea0b03b90/volumes"
Oct 30 20:07:41.609294 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:41.609262    5958 scope.go:122] "RemoveContainer" containerID="001d22446d2f6cdc068b9a38e4760b466c59cd051a865b820f94eca666a4dd85"
Oct 30 20:07:41.649175 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:41.649149    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 20:07:41.653041 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:41.653019    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 20:07:42.259582 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.259544    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"]
Oct 30 20:07:42.260219 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.260203    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c786d26b-042b-42eb-9520-434658f959d1" containerName="revoke-keys"
Oct 30 20:07:42.260279 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.260220    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c786d26b-042b-42eb-9520-434658f959d1" containerName="revoke-keys"
Oct 30 20:07:42.260355 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.260346    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c786d26b-042b-42eb-9520-434658f959d1" containerName="revoke-keys"
Oct 30 20:07:42.265166 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.265143    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:42.368507 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.368477    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f84d85\"/\"distinct-2-w3-f84d85-kserve-self-signed-certs\""
Oct 30 20:07:42.372971 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.372906    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"]
Oct 30 20:07:42.400608 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.400572    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-home\") pod \"distinct-2-w3-f84d85-kserve-5846c87596-xrrkf\" (UID: \"a1551c50-4d89-4227-838c-4644c41de49b\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:42.400917 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.400887    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-model-cache\") pod \"distinct-2-w3-f84d85-kserve-5846c87596-xrrkf\" (UID: \"a1551c50-4d89-4227-838c-4644c41de49b\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:42.401271 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.401244    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-dshm\") pod \"distinct-2-w3-f84d85-kserve-5846c87596-xrrkf\" (UID: \"a1551c50-4d89-4227-838c-4644c41de49b\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:42.401427 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.401401    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz6r5\" (UniqueName: \"kubernetes.io/projected/a1551c50-4d89-4227-838c-4644c41de49b-kube-api-access-vz6r5\") pod \"distinct-2-w3-f84d85-kserve-5846c87596-xrrkf\" (UID: \"a1551c50-4d89-4227-838c-4644c41de49b\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:42.401550 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.401445    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-tmp-dir\") pod \"distinct-2-w3-f84d85-kserve-5846c87596-xrrkf\" (UID: \"a1551c50-4d89-4227-838c-4644c41de49b\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:42.401550 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.401491    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1551c50-4d89-4227-838c-4644c41de49b-tls-certs\") pod \"distinct-2-w3-f84d85-kserve-5846c87596-xrrkf\" (UID: \"a1551c50-4d89-4227-838c-4644c41de49b\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:42.504309 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.504266    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-home\") pod \"distinct-2-w3-f84d85-kserve-5846c87596-xrrkf\" (UID: \"a1551c50-4d89-4227-838c-4644c41de49b\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:42.504575 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.504320    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-model-cache\") pod \"distinct-2-w3-f84d85-kserve-5846c87596-xrrkf\" (UID: \"a1551c50-4d89-4227-838c-4644c41de49b\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:42.504575 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.504550    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-dshm\") pod \"distinct-2-w3-f84d85-kserve-5846c87596-xrrkf\" (UID: \"a1551c50-4d89-4227-838c-4644c41de49b\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:42.504708 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.504604    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz6r5\" (UniqueName: \"kubernetes.io/projected/a1551c50-4d89-4227-838c-4644c41de49b-kube-api-access-vz6r5\") pod \"distinct-2-w3-f84d85-kserve-5846c87596-xrrkf\" (UID: \"a1551c50-4d89-4227-838c-4644c41de49b\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:42.504708 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.504645    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-tmp-dir\") pod \"distinct-2-w3-f84d85-kserve-5846c87596-xrrkf\" (UID: \"a1551c50-4d89-4227-838c-4644c41de49b\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:42.504708 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.504675    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1551c50-4d89-4227-838c-4644c41de49b-tls-certs\") pod \"distinct-2-w3-f84d85-kserve-5846c87596-xrrkf\" (UID: \"a1551c50-4d89-4227-838c-4644c41de49b\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:42.505990 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.505954    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-home\") pod \"distinct-2-w3-f84d85-kserve-5846c87596-xrrkf\" (UID: \"a1551c50-4d89-4227-838c-4644c41de49b\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:42.506243 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.506220    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-tmp-dir\") pod \"distinct-2-w3-f84d85-kserve-5846c87596-xrrkf\" (UID: \"a1551c50-4d89-4227-838c-4644c41de49b\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:42.506608 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.506585    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-model-cache\") pod \"distinct-2-w3-f84d85-kserve-5846c87596-xrrkf\" (UID: \"a1551c50-4d89-4227-838c-4644c41de49b\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:42.510101 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.508634    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1551c50-4d89-4227-838c-4644c41de49b-tls-certs\") pod \"distinct-2-w3-f84d85-kserve-5846c87596-xrrkf\" (UID: \"a1551c50-4d89-4227-838c-4644c41de49b\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:42.510101 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.509762    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-dshm\") pod \"distinct-2-w3-f84d85-kserve-5846c87596-xrrkf\" (UID: \"a1551c50-4d89-4227-838c-4644c41de49b\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:42.572651 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.572615    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz6r5\" (UniqueName: \"kubernetes.io/projected/a1551c50-4d89-4227-838c-4644c41de49b-kube-api-access-vz6r5\") pod \"distinct-2-w3-f84d85-kserve-5846c87596-xrrkf\" (UID: \"a1551c50-4d89-4227-838c-4644c41de49b\") " pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:42.577520 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.577491    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:42.966330 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:42.966288    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"]
Oct 30 20:07:42.967695 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:07:42.967667    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1551c50_4d89_4227_838c_4644c41de49b.slice/crio-97ad019c6b736b5e4d66e02294f7d65e143759def33749cc43041acbaf293d8b WatchSource:0}: Error finding container 97ad019c6b736b5e4d66e02294f7d65e143759def33749cc43041acbaf293d8b: Status 404 returned error can't find the container with id 97ad019c6b736b5e4d66e02294f7d65e143759def33749cc43041acbaf293d8b
Oct 30 20:07:43.142508 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:43.142470    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf" event={"ID":"a1551c50-4d89-4227-838c-4644c41de49b","Type":"ContainerStarted","Data":"1a22a35f291863cd1c671bcc21a1ab8893c990f2217ed611a81fba31ea4a6c91"}
Oct 30 20:07:43.142695 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:43.142541    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf" event={"ID":"a1551c50-4d89-4227-838c-4644c41de49b","Type":"ContainerStarted","Data":"97ad019c6b736b5e4d66e02294f7d65e143759def33749cc43041acbaf293d8b"}
Oct 30 20:07:43.265635 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:43.265528    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf" podStartSLOduration=1.265513641 podStartE2EDuration="1.265513641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:43.264549759 +0000 UTC m=+1501.996765152" watchObservedRunningTime="2026-10-30 20:07:43.265513641 +0000 UTC m=+1501.997729031"
Oct 30 20:07:43.578171 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:43.578142    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:47.227908 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.227870    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"]
Oct 30 20:07:47.232338 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.232318    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.343811 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.343770    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"]
Oct 30 20:07:47.376109 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.376071    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.376279 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.376202    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.376279 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.376262    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmbln\" (UniqueName: \"kubernetes.io/projected/c707594b-f9af-4025-8614-43ce0feaa1e9-kube-api-access-pmbln\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.376403 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.376322    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.376403 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.376359    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.376403 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.376396    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.376583 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.376423    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c707594b-f9af-4025-8614-43ce0feaa1e9-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.376684 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.376656    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.376752 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.376718    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.409231 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.409202    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38cbb4d-l9bhc"]
Oct 30 20:07:47.413640 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.413622    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38cbb4d-l9bhc"
Oct 30 20:07:47.467202 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.467178    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:07:47.467586 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.467566    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:07:47.477580 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.477554    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.477704 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.477598    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.477704 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.477649    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.477825 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.477706    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.477825 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.477747    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmbln\" (UniqueName: \"kubernetes.io/projected/c707594b-f9af-4025-8614-43ce0feaa1e9-kube-api-access-pmbln\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.477825 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.477806    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.478087 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.477833    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.478087 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.477896    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.478087 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.477936    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c707594b-f9af-4025-8614-43ce0feaa1e9-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.478087 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.478025    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-credential-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.478330 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.478105    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-workload-certs\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.478395 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.478346    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-data\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.478668 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.478644    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-workload-socket\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.478935 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.478911    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c707594b-f9af-4025-8614-43ce0feaa1e9-istiod-ca-cert\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.480187 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.480161    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-envoy\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.480283 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.480228    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-podinfo\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.505693 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.505661    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38cbb4d-l9bhc"]
Oct 30 20:07:47.547990 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.547935    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-token\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.548483 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.548438    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmbln\" (UniqueName: \"kubernetes.io/projected/c707594b-f9af-4025-8614-43ce0feaa1e9-kube-api-access-pmbln\") pod \"e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") " pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.549245 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.549228    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:47.579204 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.579176    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f55097-1793-4eeb-9081-ff4c6345c738-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e38cbb4d-l9bhc\" (UID: \"e9f55097-1793-4eeb-9081-ff4c6345c738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38cbb4d-l9bhc"
Oct 30 20:07:47.579300 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.579286    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f55097-1793-4eeb-9081-ff4c6345c738-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e38cbb4d-l9bhc\" (UID: \"e9f55097-1793-4eeb-9081-ff4c6345c738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38cbb4d-l9bhc"
Oct 30 20:07:47.579355 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.579341    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kg5c\" (UniqueName: \"kubernetes.io/projected/e9f55097-1793-4eeb-9081-ff4c6345c738-kube-api-access-6kg5c\") pod \"maas-api-revoke-sub-e38cbb4d-l9bhc\" (UID: \"e9f55097-1793-4eeb-9081-ff4c6345c738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38cbb4d-l9bhc"
Oct 30 20:07:47.683494 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.681261    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f55097-1793-4eeb-9081-ff4c6345c738-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e38cbb4d-l9bhc\" (UID: \"e9f55097-1793-4eeb-9081-ff4c6345c738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38cbb4d-l9bhc"
Oct 30 20:07:47.683494 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.681353    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f55097-1793-4eeb-9081-ff4c6345c738-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e38cbb4d-l9bhc\" (UID: \"e9f55097-1793-4eeb-9081-ff4c6345c738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38cbb4d-l9bhc"
Oct 30 20:07:47.683494 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.681403    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kg5c\" (UniqueName: \"kubernetes.io/projected/e9f55097-1793-4eeb-9081-ff4c6345c738-kube-api-access-6kg5c\") pod \"maas-api-revoke-sub-e38cbb4d-l9bhc\" (UID: \"e9f55097-1793-4eeb-9081-ff4c6345c738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38cbb4d-l9bhc"
Oct 30 20:07:47.683494 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.682316    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f55097-1793-4eeb-9081-ff4c6345c738-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e38cbb4d-l9bhc\" (UID: \"e9f55097-1793-4eeb-9081-ff4c6345c738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38cbb4d-l9bhc"
Oct 30 20:07:47.685035 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.685008    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f55097-1793-4eeb-9081-ff4c6345c738-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e38cbb4d-l9bhc\" (UID: \"e9f55097-1793-4eeb-9081-ff4c6345c738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38cbb4d-l9bhc"
Oct 30 20:07:47.744515 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.744400    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kg5c\" (UniqueName: \"kubernetes.io/projected/e9f55097-1793-4eeb-9081-ff4c6345c738-kube-api-access-6kg5c\") pod \"maas-api-revoke-sub-e38cbb4d-l9bhc\" (UID: \"e9f55097-1793-4eeb-9081-ff4c6345c738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38cbb4d-l9bhc"
Oct 30 20:07:47.995354 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.995292    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"]
Oct 30 20:07:47.996352 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:07:47.996329    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc707594b_f9af_4025_8614_43ce0feaa1e9.slice/crio-ee72e32027e1c685fca850ba1fe29ff746be2dc9387abf84d688e228914a1fc8 WatchSource:0}: Error finding container ee72e32027e1c685fca850ba1fe29ff746be2dc9387abf84d688e228914a1fc8: Status 404 returned error can't find the container with id ee72e32027e1c685fca850ba1fe29ff746be2dc9387abf84d688e228914a1fc8
Oct 30 20:07:47.998689 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.998659    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:07:47.998758 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.998718    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:07:47.998758 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:47.998749    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:07:48.027562 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:48.027541    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38cbb4d-l9bhc"
Oct 30 20:07:48.181769 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:48.181736    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm" event={"ID":"c707594b-f9af-4025-8614-43ce0feaa1e9","Type":"ContainerStarted","Data":"18464ae24f428ef0cc5013d5f2a7c9381ec2f15bd3027d1fc49f7bc54a700618"}
Oct 30 20:07:48.181996 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:48.181785    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm" event={"ID":"c707594b-f9af-4025-8614-43ce0feaa1e9","Type":"ContainerStarted","Data":"ee72e32027e1c685fca850ba1fe29ff746be2dc9387abf84d688e228914a1fc8"}
Oct 30 20:07:48.308214 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:48.308112    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm" podStartSLOduration=1.308097831 podStartE2EDuration="1.308097831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:48.304087531 +0000 UTC m=+1507.036302921" watchObservedRunningTime="2026-10-30 20:07:48.308097831 +0000 UTC m=+1507.040313221"
Oct 30 20:07:48.410939 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:48.410758    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38cbb4d-l9bhc"]
Oct 30 20:07:48.414242 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:07:48.414214    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f55097_1793_4eeb_9081_ff4c6345c738.slice/crio-85c355209e3e0d629e1b5299cbab20f602910096e3fdec62108b681643a6a3b3 WatchSource:0}: Error finding container 85c355209e3e0d629e1b5299cbab20f602910096e3fdec62108b681643a6a3b3: Status 404 returned error can't find the container with id 85c355209e3e0d629e1b5299cbab20f602910096e3fdec62108b681643a6a3b3
Oct 30 20:07:48.550066 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:48.550034    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:49.191028 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:49.190988    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38cbb4d-l9bhc" event={"ID":"e9f55097-1793-4eeb-9081-ff4c6345c738","Type":"ContainerStarted","Data":"9eac3ce5689a0b326b25f488d0750a731b3de4dac3884cf0da26f090de7caf73"}
Oct 30 20:07:49.191216 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:49.191052    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38cbb4d-l9bhc" event={"ID":"e9f55097-1793-4eeb-9081-ff4c6345c738","Type":"ContainerStarted","Data":"85c355209e3e0d629e1b5299cbab20f602910096e3fdec62108b681643a6a3b3"}
Oct 30 20:07:49.561129 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:49.561089    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:07:49.561614 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:49.561159    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm" podUID="c707594b-f9af-4025-8614-43ce0feaa1e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 20:07:50.199689 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:50.199651    5958 generic.go:356] "Generic (PLEG): container finished" podID="e9f55097-1793-4eeb-9081-ff4c6345c738" containerID="9eac3ce5689a0b326b25f488d0750a731b3de4dac3884cf0da26f090de7caf73" exitCode=0
Oct 30 20:07:50.199850 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:50.199736    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38cbb4d-l9bhc" event={"ID":"e9f55097-1793-4eeb-9081-ff4c6345c738","Type":"ContainerDied","Data":"9eac3ce5689a0b326b25f488d0750a731b3de4dac3884cf0da26f090de7caf73"}
Oct 30 20:07:50.561005 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:50.560836    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:07:50.561005 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:50.560915    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm" podUID="c707594b-f9af-4025-8614-43ce0feaa1e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 20:07:51.415680 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:51.415656    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38cbb4d-l9bhc"
Oct 30 20:07:51.560536 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:51.559798    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:07:51.560536 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:51.559857    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm" podUID="c707594b-f9af-4025-8614-43ce0feaa1e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 20:07:51.569974 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:51.569938    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9f55097-1793-4eeb-9081-ff4c6345c738-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f55097-1793-4eeb-9081-ff4c6345c738-maas-api-xks-ca\") pod \"e9f55097-1793-4eeb-9081-ff4c6345c738\" (UID: \"e9f55097-1793-4eeb-9081-ff4c6345c738\") "
Oct 30 20:07:51.570422 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:51.570074    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9f55097-1793-4eeb-9081-ff4c6345c738-kube-api-access-6kg5c\" (UniqueName: \"kubernetes.io/projected/e9f55097-1793-4eeb-9081-ff4c6345c738-kube-api-access-6kg5c\") pod \"e9f55097-1793-4eeb-9081-ff4c6345c738\" (UID: \"e9f55097-1793-4eeb-9081-ff4c6345c738\") "
Oct 30 20:07:51.570422 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:51.570302    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9f55097-1793-4eeb-9081-ff4c6345c738-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f55097-1793-4eeb-9081-ff4c6345c738-maas-api-openshift-ca\") pod \"e9f55097-1793-4eeb-9081-ff4c6345c738\" (UID: \"e9f55097-1793-4eeb-9081-ff4c6345c738\") "
Oct 30 20:07:51.570988 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:51.570960    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f55097-1793-4eeb-9081-ff4c6345c738-maas-api-openshift-ca" pod "e9f55097-1793-4eeb-9081-ff4c6345c738" (UID: "e9f55097-1793-4eeb-9081-ff4c6345c738"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:07:51.571141 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:51.571117    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9f55097-1793-4eeb-9081-ff4c6345c738-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:51.573009 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:51.572983    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9f55097-1793-4eeb-9081-ff4c6345c738-maas-api-xks-ca" pod "e9f55097-1793-4eeb-9081-ff4c6345c738" (UID: "e9f55097-1793-4eeb-9081-ff4c6345c738"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:07:51.577617 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:51.577589    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f55097-1793-4eeb-9081-ff4c6345c738-kube-api-access-6kg5c" pod "e9f55097-1793-4eeb-9081-ff4c6345c738" (UID: "e9f55097-1793-4eeb-9081-ff4c6345c738"). InnerVolumeSpecName "kube-api-access-6kg5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:07:51.673271 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:51.673237    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9f55097-1793-4eeb-9081-ff4c6345c738-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:51.673271 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:51.673263    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kg5c\" (UniqueName: \"kubernetes.io/projected/e9f55097-1793-4eeb-9081-ff4c6345c738-kube-api-access-6kg5c\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:52.223146 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:52.223109    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38cbb4d-l9bhc" event={"ID":"e9f55097-1793-4eeb-9081-ff4c6345c738","Type":"ContainerDied","Data":"85c355209e3e0d629e1b5299cbab20f602910096e3fdec62108b681643a6a3b3"}
Oct 30 20:07:52.223357 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:52.223150    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38cbb4d-l9bhc"
Oct 30 20:07:52.223357 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:52.223176    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85c355209e3e0d629e1b5299cbab20f602910096e3fdec62108b681643a6a3b3"
Oct 30 20:07:52.557542 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:52.556800    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:07:52.557542 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:52.556860    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm" podUID="c707594b-f9af-4025-8614-43ce0feaa1e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 20:07:52.577814 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:52.577782    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:52.591171 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:52.591145    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:53.266413 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:53.266386    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:53.560530 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:53.559784    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:07:53.560530 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:53.559855    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm" podUID="c707594b-f9af-4025-8614-43ce0feaa1e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 20:07:54.550393 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:54.550356    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 20:07:54.550815 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:54.550421    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm" podUID="c707594b-f9af-4025-8614-43ce0feaa1e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 20:07:55.554640 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:55.554608    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:55.555120 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:55.554980    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:55.555706 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:55.555684    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:07:57.735476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:57.735281    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:57.819594 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:57.819545    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" podUID="6e20c97b-66ce-485b-ba8d-90c2d9e1fa76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:57.820260 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:57.819849    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:57.890580 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:57.890532    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:57.927591 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:57.927541    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:57.948582 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:57.948534    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:58.260479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:58.250665    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf" podUID="a1551c50-4d89-4227-838c-4644c41de49b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:58.416496 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:58.392268    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:58.416496 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:58.407171    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:58.746477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:58.745697    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" podUID="6e20c97b-66ce-485b-ba8d-90c2d9e1fa76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:58.746477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:58.746053    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Oct 30 20:07:58.815500 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:58.782843    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:58.894020 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:58.893980    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 20:07:58.894742 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:58.894716    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 30 20:07:58.895239 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:58.895200    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" podUID="6e20c97b-66ce-485b-ba8d-90c2d9e1fa76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:58.895807 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:58.895790    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 20:07:58.896029 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:58.896013    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:07:58.896198 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:58.896184    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 20:07:58.946501 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:58.927625    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 20:07:58.946501 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:58.928906    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 20:07:58.977109 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:58.964680    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:58.977109 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:58.965705    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:07:58.998836 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:58.986389    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 30 20:07:59.359872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:59.233286    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:59.359872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:59.251515    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf" podUID="a1551c50-4d89-4227-838c-4644c41de49b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:59.359872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:59.354744    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:07:59.359872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:59.354820    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:59.364702 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:59.364660    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:07:59.386975 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:59.386949    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:07:59.392872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:59.392833    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:59.411567 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:59.411522    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:59.488477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:59.448328    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:59.592471 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:59.585342    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:07:59.592471 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:59.585413    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 20:07:59.592471 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:59.585444    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:07:59.592471 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:59.585494    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 20:07:59.592471 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:59.585522    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 20:07:59.628867 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:59.628787    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:07:59.630090 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:59.629866    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 20:07:59.635354 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:59.635330    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 20:07:59.641481 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:07:59.640586    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:08:04.247892 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:04.247853    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5da4abe-r54zr"]
Oct 30 20:08:04.248565 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:04.248550    5958 cpu_manager.go:414] "containerMap: removing container" podUID="e9f55097-1793-4eeb-9081-ff4c6345c738" containerName="revoke-keys"
Oct 30 20:08:04.248673 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:04.248566    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f55097-1793-4eeb-9081-ff4c6345c738" containerName="revoke-keys"
Oct 30 20:08:04.248673 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:04.248661    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f55097-1793-4eeb-9081-ff4c6345c738" containerName="revoke-keys"
Oct 30 20:08:04.252393 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:04.252375    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da4abe-r54zr"
Oct 30 20:08:04.355736 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:04.355705    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:08:04.355893 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:04.355743    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:08:04.359344 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:04.359309    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5da4abe-r54zr"]
Oct 30 20:08:04.382750 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:04.382726    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d82aa63-2171-4e98-8861-aedf321fd81f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5da4abe-r54zr\" (UID: \"2d82aa63-2171-4e98-8861-aedf321fd81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da4abe-r54zr"
Oct 30 20:08:04.382930 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:04.382910    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjnkn\" (UniqueName: \"kubernetes.io/projected/2d82aa63-2171-4e98-8861-aedf321fd81f-kube-api-access-bjnkn\") pod \"maas-api-revoke-sub-e5da4abe-r54zr\" (UID: \"2d82aa63-2171-4e98-8861-aedf321fd81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da4abe-r54zr"
Oct 30 20:08:04.383127 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:04.383107    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d82aa63-2171-4e98-8861-aedf321fd81f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5da4abe-r54zr\" (UID: \"2d82aa63-2171-4e98-8861-aedf321fd81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da4abe-r54zr"
Oct 30 20:08:04.484656 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:04.484616    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d82aa63-2171-4e98-8861-aedf321fd81f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5da4abe-r54zr\" (UID: \"2d82aa63-2171-4e98-8861-aedf321fd81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da4abe-r54zr"
Oct 30 20:08:04.484839 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:04.484710    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d82aa63-2171-4e98-8861-aedf321fd81f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5da4abe-r54zr\" (UID: \"2d82aa63-2171-4e98-8861-aedf321fd81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da4abe-r54zr"
Oct 30 20:08:04.484939 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:04.484903    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjnkn\" (UniqueName: \"kubernetes.io/projected/2d82aa63-2171-4e98-8861-aedf321fd81f-kube-api-access-bjnkn\") pod \"maas-api-revoke-sub-e5da4abe-r54zr\" (UID: \"2d82aa63-2171-4e98-8861-aedf321fd81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da4abe-r54zr"
Oct 30 20:08:04.485440 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:04.485414    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d82aa63-2171-4e98-8861-aedf321fd81f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5da4abe-r54zr\" (UID: \"2d82aa63-2171-4e98-8861-aedf321fd81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da4abe-r54zr"
Oct 30 20:08:04.487372 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:04.487350    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d82aa63-2171-4e98-8861-aedf321fd81f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5da4abe-r54zr\" (UID: \"2d82aa63-2171-4e98-8861-aedf321fd81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da4abe-r54zr"
Oct 30 20:08:04.546406 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:04.546343    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjnkn\" (UniqueName: \"kubernetes.io/projected/2d82aa63-2171-4e98-8861-aedf321fd81f-kube-api-access-bjnkn\") pod \"maas-api-revoke-sub-e5da4abe-r54zr\" (UID: \"2d82aa63-2171-4e98-8861-aedf321fd81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da4abe-r54zr"
Oct 30 20:08:04.565705 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:04.565680    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da4abe-r54zr"
Oct 30 20:08:04.944082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:04.944048    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5da4abe-r54zr"]
Oct 30 20:08:04.944327 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:08:04.944300    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d82aa63_2171_4e98_8861_aedf321fd81f.slice/crio-c5e5ee4dcb5d141a789be4dedd0e20aa30d07838841bab64c51123e6688d23ed WatchSource:0}: Error finding container c5e5ee4dcb5d141a789be4dedd0e20aa30d07838841bab64c51123e6688d23ed: Status 404 returned error can't find the container with id c5e5ee4dcb5d141a789be4dedd0e20aa30d07838841bab64c51123e6688d23ed
Oct 30 20:08:05.848965 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:05.848931    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da4abe-r54zr" event={"ID":"2d82aa63-2171-4e98-8861-aedf321fd81f","Type":"ContainerStarted","Data":"74f5ec1e219c06a3ae044066e7185de093f3414f7886d01a6d3019b1101a6d07"}
Oct 30 20:08:05.849333 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:05.848990    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da4abe-r54zr" event={"ID":"2d82aa63-2171-4e98-8861-aedf321fd81f","Type":"ContainerStarted","Data":"c5e5ee4dcb5d141a789be4dedd0e20aa30d07838841bab64c51123e6688d23ed"}
Oct 30 20:08:05.968367 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:05.968317    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da4abe-r54zr" podStartSLOduration=1.968302733 podStartE2EDuration="1.968302733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:05.965516634 +0000 UTC m=+1524.697732023" watchObservedRunningTime="2026-10-30 20:08:05.968302733 +0000 UTC m=+1524.700518122"
Oct 30 20:08:06.607338 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:08:06.607295    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:08:06.858264 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:06.858188    5958 generic.go:356] "Generic (PLEG): container finished" podID="2d82aa63-2171-4e98-8861-aedf321fd81f" containerID="74f5ec1e219c06a3ae044066e7185de093f3414f7886d01a6d3019b1101a6d07" exitCode=0
Oct 30 20:08:06.858639 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:06.858272    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da4abe-r54zr" event={"ID":"2d82aa63-2171-4e98-8861-aedf321fd81f","Type":"ContainerDied","Data":"74f5ec1e219c06a3ae044066e7185de093f3414f7886d01a6d3019b1101a6d07"}
Oct 30 20:08:07.582704 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:07.582661    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"]
Oct 30 20:08:07.582917 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:07.582890    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl" podUID="3e3b7c1d-c991-4375-8caa-df21545e1fd8" containerName="istio-proxy" containerID="cri-o://8f096bee4e36f6a8e26b25492fea44706b34ea384c3d6c44fa2783fed74f1a8c" gracePeriod=30
Oct 30 20:08:08.035596 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:08.035568    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da4abe-r54zr"
Oct 30 20:08:08.137082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:08.137022    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d82aa63-2171-4e98-8861-aedf321fd81f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d82aa63-2171-4e98-8861-aedf321fd81f-maas-api-openshift-ca\") pod \"2d82aa63-2171-4e98-8861-aedf321fd81f\" (UID: \"2d82aa63-2171-4e98-8861-aedf321fd81f\") "
Oct 30 20:08:08.137233 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:08.137219    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d82aa63-2171-4e98-8861-aedf321fd81f-kube-api-access-bjnkn\" (UniqueName: \"kubernetes.io/projected/2d82aa63-2171-4e98-8861-aedf321fd81f-kube-api-access-bjnkn\") pod \"2d82aa63-2171-4e98-8861-aedf321fd81f\" (UID: \"2d82aa63-2171-4e98-8861-aedf321fd81f\") "
Oct 30 20:08:08.137283 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:08.137268    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d82aa63-2171-4e98-8861-aedf321fd81f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d82aa63-2171-4e98-8861-aedf321fd81f-maas-api-xks-ca\") pod \"2d82aa63-2171-4e98-8861-aedf321fd81f\" (UID: \"2d82aa63-2171-4e98-8861-aedf321fd81f\") "
Oct 30 20:08:08.137327 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:08.137306    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d82aa63-2171-4e98-8861-aedf321fd81f-maas-api-openshift-ca" pod "2d82aa63-2171-4e98-8861-aedf321fd81f" (UID: "2d82aa63-2171-4e98-8861-aedf321fd81f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:08:08.137831 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:08.137804    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d82aa63-2171-4e98-8861-aedf321fd81f-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:08.139415 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:08.139385    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d82aa63-2171-4e98-8861-aedf321fd81f-maas-api-xks-ca" pod "2d82aa63-2171-4e98-8861-aedf321fd81f" (UID: "2d82aa63-2171-4e98-8861-aedf321fd81f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:08:08.139522 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:08.139495    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d82aa63-2171-4e98-8861-aedf321fd81f-kube-api-access-bjnkn" pod "2d82aa63-2171-4e98-8861-aedf321fd81f" (UID: "2d82aa63-2171-4e98-8861-aedf321fd81f"). InnerVolumeSpecName "kube-api-access-bjnkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:08.238929 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:08.238900    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjnkn\" (UniqueName: \"kubernetes.io/projected/2d82aa63-2171-4e98-8861-aedf321fd81f-kube-api-access-bjnkn\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:08.238929 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:08.238931    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d82aa63-2171-4e98-8861-aedf321fd81f-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:08.873437 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:08.873392    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da4abe-r54zr" event={"ID":"2d82aa63-2171-4e98-8861-aedf321fd81f","Type":"ContainerDied","Data":"c5e5ee4dcb5d141a789be4dedd0e20aa30d07838841bab64c51123e6688d23ed"}
Oct 30 20:08:08.873437 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:08.873439    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5e5ee4dcb5d141a789be4dedd0e20aa30d07838841bab64c51123e6688d23ed"
Oct 30 20:08:08.873437 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:08.873400    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da4abe-r54zr"
Oct 30 20:08:12.878866 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.878837    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:08:12.903298 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.903266    5958 generic.go:356] "Generic (PLEG): container finished" podID="3e3b7c1d-c991-4375-8caa-df21545e1fd8" containerID="8f096bee4e36f6a8e26b25492fea44706b34ea384c3d6c44fa2783fed74f1a8c" exitCode=0
Oct 30 20:08:12.903424 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.903325    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"
Oct 30 20:08:12.903424 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.903347    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl" event={"ID":"3e3b7c1d-c991-4375-8caa-df21545e1fd8","Type":"ContainerDied","Data":"8f096bee4e36f6a8e26b25492fea44706b34ea384c3d6c44fa2783fed74f1a8c"}
Oct 30 20:08:12.903553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.903435    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl" event={"ID":"3e3b7c1d-c991-4375-8caa-df21545e1fd8","Type":"ContainerDied","Data":"74e492458dc73520a63c4bc834e5655d8ce0954c67fb75b40fe6536d989864e9"}
Oct 30 20:08:12.903553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.903477    5958 scope.go:122] "RemoveContainer" containerID="8f096bee4e36f6a8e26b25492fea44706b34ea384c3d6c44fa2783fed74f1a8c"
Oct 30 20:08:12.916807 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.916787    5958 scope.go:122] "RemoveContainer" containerID="8f096bee4e36f6a8e26b25492fea44706b34ea384c3d6c44fa2783fed74f1a8c"
Oct 30 20:08:12.917088 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:08:12.917065    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f096bee4e36f6a8e26b25492fea44706b34ea384c3d6c44fa2783fed74f1a8c\": container with ID starting with 8f096bee4e36f6a8e26b25492fea44706b34ea384c3d6c44fa2783fed74f1a8c not found: ID does not exist" containerID="8f096bee4e36f6a8e26b25492fea44706b34ea384c3d6c44fa2783fed74f1a8c"
Oct 30 20:08:12.917177 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.917096    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f096bee4e36f6a8e26b25492fea44706b34ea384c3d6c44fa2783fed74f1a8c"} err="failed to get container status \"8f096bee4e36f6a8e26b25492fea44706b34ea384c3d6c44fa2783fed74f1a8c\": rpc error: code = NotFound desc = could not find container \"8f096bee4e36f6a8e26b25492fea44706b34ea384c3d6c44fa2783fed74f1a8c\": container with ID starting with 8f096bee4e36f6a8e26b25492fea44706b34ea384c3d6c44fa2783fed74f1a8c not found: ID does not exist"
Oct 30 20:08:12.918595 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.918570    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-token\" (UniqueName: \"kubernetes.io/projected/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-token\") pod \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") "
Oct 30 20:08:12.918673 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.918618    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-workload-socket\") pod \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") "
Oct 30 20:08:12.918732 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.918683    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-workload-certs\") pod \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") "
Oct 30 20:08:12.918787 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.918737    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e3b7c1d-c991-4375-8caa-df21545e1fd8-kube-api-access-kfdt9\" (UniqueName: \"kubernetes.io/projected/3e3b7c1d-c991-4375-8caa-df21545e1fd8-kube-api-access-kfdt9\") pod \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") "
Oct 30 20:08:12.918841 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.918818    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-credential-socket\") pod \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") "
Oct 30 20:08:12.918899 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.918850    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-envoy\") pod \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") "
Oct 30 20:08:12.918899 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.918879    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-data\") pod \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") "
Oct 30 20:08:12.918995 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.918930    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-workload-socket" pod "3e3b7c1d-c991-4375-8caa-df21545e1fd8" (UID: "3e3b7c1d-c991-4375-8caa-df21545e1fd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:12.919047 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.919026    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-podinfo\") pod \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") "
Oct 30 20:08:12.919106 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.919067    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istiod-ca-cert\") pod \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\" (UID: \"3e3b7c1d-c991-4375-8caa-df21545e1fd8\") "
Oct 30 20:08:12.919106 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.919073    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-workload-certs" pod "3e3b7c1d-c991-4375-8caa-df21545e1fd8" (UID: "3e3b7c1d-c991-4375-8caa-df21545e1fd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:12.919236 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.919212    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-data" pod "3e3b7c1d-c991-4375-8caa-df21545e1fd8" (UID: "3e3b7c1d-c991-4375-8caa-df21545e1fd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:12.919469 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.919432    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-credential-socket" pod "3e3b7c1d-c991-4375-8caa-df21545e1fd8" (UID: "3e3b7c1d-c991-4375-8caa-df21545e1fd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:12.919592 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.919559    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istiod-ca-cert" pod "3e3b7c1d-c991-4375-8caa-df21545e1fd8" (UID: "3e3b7c1d-c991-4375-8caa-df21545e1fd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:08:12.919842 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.919817    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:12.919984 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.919844    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:12.919984 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.919859    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:12.919984 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.919873    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:12.919984 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.919886    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:12.921208 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.921184    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-token" pod "3e3b7c1d-c991-4375-8caa-df21545e1fd8" (UID: "3e3b7c1d-c991-4375-8caa-df21545e1fd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:12.921303 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.921242    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-envoy" pod "3e3b7c1d-c991-4375-8caa-df21545e1fd8" (UID: "3e3b7c1d-c991-4375-8caa-df21545e1fd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:08:12.921379 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.921362    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-podinfo" pod "3e3b7c1d-c991-4375-8caa-df21545e1fd8" (UID: "3e3b7c1d-c991-4375-8caa-df21545e1fd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:08:12.922035 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:12.922015    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e3b7c1d-c991-4375-8caa-df21545e1fd8-kube-api-access-kfdt9" pod "3e3b7c1d-c991-4375-8caa-df21545e1fd8" (UID: "3e3b7c1d-c991-4375-8caa-df21545e1fd8"). InnerVolumeSpecName "kube-api-access-kfdt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:13.020979 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:13.020950    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:13.020979 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:13.020979    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfdt9\" (UniqueName: \"kubernetes.io/projected/3e3b7c1d-c991-4375-8caa-df21545e1fd8-kube-api-access-kfdt9\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:13.021137 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:13.020989    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:13.021137 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:13.020999    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e3b7c1d-c991-4375-8caa-df21545e1fd8-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:13.331296 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:13.331258    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"]
Oct 30 20:08:13.338061 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:13.338030    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xf8pl"]
Oct 30 20:08:13.399947 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:13.399916    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e3b7c1d-c991-4375-8caa-df21545e1fd8" path="/var/lib/kubelet/pods/3e3b7c1d-c991-4375-8caa-df21545e1fd8/volumes"
Oct 30 20:08:30.366629 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:30.366565    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 30 20:08:30.403925 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:30.400671    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 30 20:08:30.403925 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:30.401144    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 20:08:30.533702 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:30.533238    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf" podUID="a1551c50-4d89-4227-838c-4644c41de49b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:30.570654 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:30.570585    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:30.618745 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:30.618614    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:30.637973 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:30.637892    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:30.639752 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:30.639705    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" podUID="6e20c97b-66ce-485b-ba8d-90c2d9e1fa76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:30.640273 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:30.640248    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 20:08:30.640792 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:30.640768    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:30.674359 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:30.673561    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 20:08:30.679922 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:30.679209    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:30.953553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:30.951574    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:08:30.953553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:30.951644    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5" podUID="0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:31.378477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:31.376973    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:31.378477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:31.377836    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:08:31.402023 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:31.401983    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:31.402658 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:31.402611    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 30 20:08:31.463875 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:31.463836    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 20:08:31.464174 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:31.464157    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:08:31.546503 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:31.543941    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf" podUID="a1551c50-4d89-4227-838c-4644c41de49b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:31.546503 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:31.544550    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:08:31.650210 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:31.650120    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" podUID="6e20c97b-66ce-485b-ba8d-90c2d9e1fa76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded"
Oct 30 20:08:31.685202 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:31.685156    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 20:08:32.027504 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:31.685488    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:32.028169 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:31.736582    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 20:08:32.028572 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:31.649920    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 20:08:32.032517 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:31.736610    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:32.057493 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:31.748623    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 20:08:32.085507 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:31.809964    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 20:08:32.085507 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:32.059061    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 20:08:32.085507 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:32.059107    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:08:32.085507 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:32.059168    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 20:08:32.085507 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:32.059207    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:08:32.085507 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:32.059240    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:08:32.085507 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:32.059274    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:08:32.395289 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:32.384048    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:32.432499 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:32.403546    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:32.433804 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:32.403584    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 20:08:32.570216 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:32.570170    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf" podUID="a1551c50-4d89-4227-838c-4644c41de49b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:32.671526 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:32.666772    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" podUID="6e20c97b-66ce-485b-ba8d-90c2d9e1fa76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:32.858478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:32.822036    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded"
Oct 30 20:08:32.862532 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:32.858791    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:33.041889 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:33.041801    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"]
Oct 30 20:08:33.170476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:33.168360    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:33.207481 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:33.184534    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:33.237143 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:33.208073    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 30 20:08:33.237143 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:33.208109    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 20:08:33.237143 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:33.208134    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 20:08:33.237143 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:33.208163    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 20:08:33.419597 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:33.419567    5958 cpu_manager.go:414] "containerMap: removing container" podUID="2d82aa63-2171-4e98-8861-aedf321fd81f" containerName="revoke-keys"
Oct 30 20:08:33.427082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:33.427042    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d82aa63-2171-4e98-8861-aedf321fd81f" containerName="revoke-keys"
Oct 30 20:08:33.447044 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:33.447013    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3e3b7c1d-c991-4375-8caa-df21545e1fd8" containerName="istio-proxy"
Oct 30 20:08:33.447258 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:33.447243    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e3b7c1d-c991-4375-8caa-df21545e1fd8" containerName="istio-proxy"
Oct 30 20:08:33.479512 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:33.447663    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d82aa63-2171-4e98-8861-aedf321fd81f" containerName="revoke-keys"
Oct 30 20:08:33.479805 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:33.479777    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e3b7c1d-c991-4375-8caa-df21545e1fd8" containerName="istio-proxy"
Oct 30 20:08:33.482148 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:33.427336    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:33.482708 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:33.442902    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 30 20:08:33.483108 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:33.442772    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:33.549245 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:33.549207    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:08:33.549954 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:33.549613    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw" podUID="56ec387e-8e34-472e-b8f5-032a57c517e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:33.571595 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:33.571548    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf" podUID="a1551c50-4d89-4227-838c-4644c41de49b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:33.711533 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:33.703501    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" podUID="6e20c97b-66ce-485b-ba8d-90c2d9e1fa76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:33.760970 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:33.760920    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf" podUID="a1551c50-4d89-4227-838c-4644c41de49b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/health\": context deadline exceeded"
Oct 30 20:08:33.919748 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:33.919710    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.173611 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.173569    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:34.239090 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.217583    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.239090 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.217676    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.239090 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.217775    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42ccg\" (UniqueName: \"kubernetes.io/projected/83b30dc9-50cf-4d71-9d59-b37e1e19f971-kube-api-access-42ccg\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.239090 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.217805    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.239090 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.217835    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.239090 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.217910    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.239090 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.218057    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.239090 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.218153    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.239090 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.218241    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.281772 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.281732    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:34.303688 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.291025    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 30 20:08:34.303688 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.302545    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 20:08:34.305039 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.303983    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 20:08:34.312474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.306820    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:34.353202 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.353167    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"]
Oct 30 20:08:34.384535 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.353326    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.384891 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.353813    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-workload-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.516534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.385572    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.516534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.386083    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.516534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.386162    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.516534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.386229    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.516534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.386329    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42ccg\" (UniqueName: \"kubernetes.io/projected/83b30dc9-50cf-4d71-9d59-b37e1e19f971-kube-api-access-42ccg\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.516534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.386358    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.516534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.386383    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.516534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.386480    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.516534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.387316    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istiod-ca-cert\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.516534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.387624    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-workload-certs\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.516534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.387877    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-data\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.516534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.420386    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-credential-socket\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.516534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.420933    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-podinfo\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.516534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.450216    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-envoy\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.516534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.489875    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:34.516534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.491050    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 30 20:08:34.516534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.492862    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 30 20:08:34.667393 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.593653    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf" podUID="a1551c50-4d89-4227-838c-4644c41de49b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:34.757479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.746039    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42ccg\" (UniqueName: \"kubernetes.io/projected/83b30dc9-50cf-4d71-9d59-b37e1e19f971-kube-api-access-42ccg\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:34.787103 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.766481    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:34.787103 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.767058    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" podUID="6e20c97b-66ce-485b-ba8d-90c2d9e1fa76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded"
Oct 30 20:08:34.935597 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:34.935548    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:35.003473 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:35.003402    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-token\") pod \"e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") " pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:35.194490 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:35.190847    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:35.223489 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:35.210299    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:35.336850 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:35.306156    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:35.336850 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:35.306571    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 20:08:35.336850 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:35.306887    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 30 20:08:35.336850 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:35.311770    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 20:08:35.336850 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:35.315832    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:35.510119 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:35.509111    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:35.510119 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:35.509563    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 20:08:35.510119 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:35.509874    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 30 20:08:35.597300 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:35.597245    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf" podUID="a1551c50-4d89-4227-838c-4644c41de49b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:35.775373 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:35.775267    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" podUID="6e20c97b-66ce-485b-ba8d-90c2d9e1fa76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:36.044538 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.028639    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:08:36.044538 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.028703    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6" podUID="6bc60e1a-391d-43b0-8b8e-2f6f33c234be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:36.200013 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.199966    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 20:08:36.309378 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.309278    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:36.336474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.311499    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:36.382264 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.337697    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 20:08:36.387854 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.337743    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:36.387854 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.362588    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Oct 30 20:08:36.557389 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.527295    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:36.558071 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.558019    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 30 20:08:36.558443 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.558419    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 20:08:36.611481 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.598265    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf" podUID="a1551c50-4d89-4227-838c-4644c41de49b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:08:36.805534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.800207    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" podUID="6e20c97b-66ce-485b-ba8d-90c2d9e1fa76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded"
Oct 30 20:08:36.901747 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.901710    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 20:08:36.906117 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.906093    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"]
Oct 30 20:08:36.961307 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.961267    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:08:36.963533 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.963489    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:08:36.971784 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.971747    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:08:36.982722 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.979097    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:08:36.982722 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.965797    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:08:36.982722 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.979668    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 20:08:36.982722 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.979705    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:08:36.982722 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.979738    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:08:36.984129 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:08:36.984097    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:08:36.989001 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.988674    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:08:36.993014 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.992940    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:08:36.993693 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.993657    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 20:08:36.994299 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.994281    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 20:08:36.997497 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.997445    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:08:36.998371 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:36.998340    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597" event={"ID":"83b30dc9-50cf-4d71-9d59-b37e1e19f971","Type":"ContainerStarted","Data":"6f8f3f8f437e1eb3fb717fe18455d523a313d860d4a37d7824bae687579ba04f"}
Oct 30 20:08:38.010623 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:38.010586    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597" event={"ID":"83b30dc9-50cf-4d71-9d59-b37e1e19f971","Type":"ContainerStarted","Data":"51114a8cf5ae3c66afea5a8d31b2050e87403eaa25266a7e3781e58a71c77cbb"}
Oct 30 20:08:38.211042 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:38.211007    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:38.212262 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:38.212232    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 20:08:38.212397 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:38.212290    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597" podUID="83b30dc9-50cf-4d71-9d59-b37e1e19f971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 20:08:38.431825 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:38.431770    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597" podStartSLOduration=6.4317543839999995 podStartE2EDuration="6.431754384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:38.427295703 +0000 UTC m=+1557.159511104" watchObservedRunningTime="2026-10-30 20:08:38.431754384 +0000 UTC m=+1557.163969773"
Oct 30 20:08:39.211436 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:39.211401    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 20:08:39.211820 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:39.211469    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597" podUID="83b30dc9-50cf-4d71-9d59-b37e1e19f971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 20:08:40.211089 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:40.211056    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 20:08:40.211262 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:40.211125    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597" podUID="83b30dc9-50cf-4d71-9d59-b37e1e19f971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 20:08:41.211405 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:41.211359    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 20:08:41.211873 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:41.211432    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597" podUID="83b30dc9-50cf-4d71-9d59-b37e1e19f971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 20:08:41.654114 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:41.654079    5958 scope.go:122] "RemoveContainer" containerID="9861fc86be57f0db704d78e824603a7a91bb8bdefa710dea8bee8860ee9df441"
Oct 30 20:08:41.669701 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:41.669677    5958 scope.go:122] "RemoveContainer" containerID="551eb85482cd13f3ecfc0c22722f772f31e0dad065c3b2125d657effb2f4b163"
Oct 30 20:08:42.211881 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:42.211844    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 20:08:42.212349 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:42.211908    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597" podUID="83b30dc9-50cf-4d71-9d59-b37e1e19f971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 20:08:43.211812 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:43.211771    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 20:08:43.211966 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:43.211845    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597" podUID="83b30dc9-50cf-4d71-9d59-b37e1e19f971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 20:08:43.677794 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:43.677759    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8096ef7-k4w6w"]
Oct 30 20:08:43.684081 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:43.684058    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8096ef7-k4w6w"
Oct 30 20:08:43.738181 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:43.738142    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:08:43.743823 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:43.743798    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8096ef7-k4w6w"]
Oct 30 20:08:43.787312 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:43.787289    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:08:43.791307 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:43.791277    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jclmz\" (UniqueName: \"kubernetes.io/projected/be91550e-f729-4788-8e76-70aea3d869c6-kube-api-access-jclmz\") pod \"maas-api-revoke-sub-c8096ef7-k4w6w\" (UID: \"be91550e-f729-4788-8e76-70aea3d869c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8096ef7-k4w6w"
Oct 30 20:08:43.791534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:43.791510    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be91550e-f729-4788-8e76-70aea3d869c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8096ef7-k4w6w\" (UID: \"be91550e-f729-4788-8e76-70aea3d869c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8096ef7-k4w6w"
Oct 30 20:08:43.791729 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:43.791710    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be91550e-f729-4788-8e76-70aea3d869c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8096ef7-k4w6w\" (UID: \"be91550e-f729-4788-8e76-70aea3d869c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8096ef7-k4w6w"
Oct 30 20:08:43.893604 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:43.893572    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be91550e-f729-4788-8e76-70aea3d869c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8096ef7-k4w6w\" (UID: \"be91550e-f729-4788-8e76-70aea3d869c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8096ef7-k4w6w"
Oct 30 20:08:43.893826 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:43.893716    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jclmz\" (UniqueName: \"kubernetes.io/projected/be91550e-f729-4788-8e76-70aea3d869c6-kube-api-access-jclmz\") pod \"maas-api-revoke-sub-c8096ef7-k4w6w\" (UID: \"be91550e-f729-4788-8e76-70aea3d869c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8096ef7-k4w6w"
Oct 30 20:08:43.893894 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:43.893856    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be91550e-f729-4788-8e76-70aea3d869c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8096ef7-k4w6w\" (UID: \"be91550e-f729-4788-8e76-70aea3d869c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8096ef7-k4w6w"
Oct 30 20:08:43.894738 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:43.894714    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be91550e-f729-4788-8e76-70aea3d869c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8096ef7-k4w6w\" (UID: \"be91550e-f729-4788-8e76-70aea3d869c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8096ef7-k4w6w"
Oct 30 20:08:43.897728 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:43.897703    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be91550e-f729-4788-8e76-70aea3d869c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8096ef7-k4w6w\" (UID: \"be91550e-f729-4788-8e76-70aea3d869c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8096ef7-k4w6w"
Oct 30 20:08:43.955375 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:43.955311    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jclmz\" (UniqueName: \"kubernetes.io/projected/be91550e-f729-4788-8e76-70aea3d869c6-kube-api-access-jclmz\") pod \"maas-api-revoke-sub-c8096ef7-k4w6w\" (UID: \"be91550e-f729-4788-8e76-70aea3d869c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8096ef7-k4w6w"
Oct 30 20:08:44.002311 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:44.002282    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8096ef7-k4w6w"
Oct 30 20:08:44.211927 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:44.211841    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 20:08:44.212099 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:44.211935    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597" podUID="83b30dc9-50cf-4d71-9d59-b37e1e19f971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 20:08:44.391433 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:44.390724    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8096ef7-k4w6w"]
Oct 30 20:08:44.398758 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:08:44.398727    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe91550e_f729_4788_8e76_70aea3d869c6.slice/crio-e060804dc97135ddd414af40f4cdd83e22fa94632afeffcd76bc4003b0ac06a9 WatchSource:0}: Error finding container e060804dc97135ddd414af40f4cdd83e22fa94632afeffcd76bc4003b0ac06a9: Status 404 returned error can't find the container with id e060804dc97135ddd414af40f4cdd83e22fa94632afeffcd76bc4003b0ac06a9
Oct 30 20:08:45.092868 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:45.092833    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8096ef7-k4w6w" event={"ID":"be91550e-f729-4788-8e76-70aea3d869c6","Type":"ContainerStarted","Data":"359b5396893b6ac99aae26a70cedd3eae2191b01e7f6530a2ff5cba1e93a992b"}
Oct 30 20:08:45.093021 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:45.092889    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8096ef7-k4w6w" event={"ID":"be91550e-f729-4788-8e76-70aea3d869c6","Type":"ContainerStarted","Data":"e060804dc97135ddd414af40f4cdd83e22fa94632afeffcd76bc4003b0ac06a9"}
Oct 30 20:08:45.215426 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:45.215389    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:45.215811 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:45.215691    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:45.216293 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:45.216273    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:08:45.253807 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:45.253757    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8096ef7-k4w6w" podStartSLOduration=2.253745233 podStartE2EDuration="2.253745233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:45.25299357 +0000 UTC m=+1563.985208958" watchObservedRunningTime="2026-10-30 20:08:45.253745233 +0000 UTC m=+1563.985960645"
Oct 30 20:08:46.102101 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:46.102061    5958 generic.go:356] "Generic (PLEG): container finished" podID="be91550e-f729-4788-8e76-70aea3d869c6" containerID="359b5396893b6ac99aae26a70cedd3eae2191b01e7f6530a2ff5cba1e93a992b" exitCode=0
Oct 30 20:08:46.102254 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:46.102137    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8096ef7-k4w6w" event={"ID":"be91550e-f729-4788-8e76-70aea3d869c6","Type":"ContainerDied","Data":"359b5396893b6ac99aae26a70cedd3eae2191b01e7f6530a2ff5cba1e93a992b"}
Oct 30 20:08:47.279360 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:47.279329    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8096ef7-k4w6w"
Oct 30 20:08:47.445989 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:47.445906    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be91550e-f729-4788-8e76-70aea3d869c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be91550e-f729-4788-8e76-70aea3d869c6-maas-api-xks-ca\") pod \"be91550e-f729-4788-8e76-70aea3d869c6\" (UID: \"be91550e-f729-4788-8e76-70aea3d869c6\") "
Oct 30 20:08:47.446156 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:47.446015    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be91550e-f729-4788-8e76-70aea3d869c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be91550e-f729-4788-8e76-70aea3d869c6-maas-api-openshift-ca\") pod \"be91550e-f729-4788-8e76-70aea3d869c6\" (UID: \"be91550e-f729-4788-8e76-70aea3d869c6\") "
Oct 30 20:08:47.446214 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:47.446196    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be91550e-f729-4788-8e76-70aea3d869c6-kube-api-access-jclmz\" (UniqueName: \"kubernetes.io/projected/be91550e-f729-4788-8e76-70aea3d869c6-kube-api-access-jclmz\") pod \"be91550e-f729-4788-8e76-70aea3d869c6\" (UID: \"be91550e-f729-4788-8e76-70aea3d869c6\") "
Oct 30 20:08:47.448260 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:47.448192    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be91550e-f729-4788-8e76-70aea3d869c6-maas-api-openshift-ca" pod "be91550e-f729-4788-8e76-70aea3d869c6" (UID: "be91550e-f729-4788-8e76-70aea3d869c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:08:47.450658 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:47.450623    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be91550e-f729-4788-8e76-70aea3d869c6-maas-api-xks-ca" pod "be91550e-f729-4788-8e76-70aea3d869c6" (UID: "be91550e-f729-4788-8e76-70aea3d869c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:08:47.455496 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:47.455470    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be91550e-f729-4788-8e76-70aea3d869c6-kube-api-access-jclmz" pod "be91550e-f729-4788-8e76-70aea3d869c6" (UID: "be91550e-f729-4788-8e76-70aea3d869c6"). InnerVolumeSpecName "kube-api-access-jclmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:47.549081 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:47.549049    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jclmz\" (UniqueName: \"kubernetes.io/projected/be91550e-f729-4788-8e76-70aea3d869c6-kube-api-access-jclmz\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:47.549081 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:47.549078    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be91550e-f729-4788-8e76-70aea3d869c6-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:47.549081 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:47.549087    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be91550e-f729-4788-8e76-70aea3d869c6-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:48.054149 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.054119    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"]
Oct 30 20:08:48.054812 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.054797    5958 cpu_manager.go:414] "containerMap: removing container" podUID="be91550e-f729-4788-8e76-70aea3d869c6" containerName="revoke-keys"
Oct 30 20:08:48.054876 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.054814    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be91550e-f729-4788-8e76-70aea3d869c6" containerName="revoke-keys"
Oct 30 20:08:48.054981 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.054971    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="be91550e-f729-4788-8e76-70aea3d869c6" containerName="revoke-keys"
Oct 30 20:08:48.059040 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.059014    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.119539 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.119509    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8096ef7-k4w6w"
Oct 30 20:08:48.119681 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.119514    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8096ef7-k4w6w" event={"ID":"be91550e-f729-4788-8e76-70aea3d869c6","Type":"ContainerDied","Data":"e060804dc97135ddd414af40f4cdd83e22fa94632afeffcd76bc4003b0ac06a9"}
Oct 30 20:08:48.119681 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.119638    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e060804dc97135ddd414af40f4cdd83e22fa94632afeffcd76bc4003b0ac06a9"
Oct 30 20:08:48.153218 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.153185    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"]
Oct 30 20:08:48.157038 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.157003    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.157178 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.157116    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdqfz\" (UniqueName: \"kubernetes.io/projected/3017938e-ba45-4d82-b5bb-15919e2b413e-kube-api-access-xdqfz\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.157178 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.157148    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.157297 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.157240    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.157354 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.157304    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3017938e-ba45-4d82-b5bb-15919e2b413e-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.157408 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.157356    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.157408 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.157392    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.157567 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.157431    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.157616 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.157593    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.259408 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.259373    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3017938e-ba45-4d82-b5bb-15919e2b413e-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.259620 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.259433    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.259620 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.259611    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.259740 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.259645    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.259792 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.259763    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.259862 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.259847    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.259943 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.259927    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdqfz\" (UniqueName: \"kubernetes.io/projected/3017938e-ba45-4d82-b5bb-15919e2b413e-kube-api-access-xdqfz\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.259999 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.259957    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.260055 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.260047    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.260127 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.260086    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-workload-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.260362 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.260337    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3017938e-ba45-4d82-b5bb-15919e2b413e-istiod-ca-cert\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.261079 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.261051    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-data\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.261403 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.261338    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-credential-socket\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.261403 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.261384    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-workload-certs\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.262606 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.262557    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-podinfo\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.263552 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.263525    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-envoy\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.321362 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.321333    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdqfz\" (UniqueName: \"kubernetes.io/projected/3017938e-ba45-4d82-b5bb-15919e2b413e-kube-api-access-xdqfz\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.321986 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.321959    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-token\") pod \"e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") " pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.375267 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.375233    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:48.816392 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.816344    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"]
Oct 30 20:08:48.818617 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:08:48.818587    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3017938e_ba45_4d82_b5bb_15919e2b413e.slice/crio-7fef5f007078a6f6c6a9478fd71c8ac293e1f42d7d4942a2be8644227227de6e WatchSource:0}: Error finding container 7fef5f007078a6f6c6a9478fd71c8ac293e1f42d7d4942a2be8644227227de6e: Status 404 returned error can't find the container with id 7fef5f007078a6f6c6a9478fd71c8ac293e1f42d7d4942a2be8644227227de6e
Oct 30 20:08:48.821208 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.821174    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:08:48.821294 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.821243    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:08:48.821294 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:48.821270    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:08:49.039488 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.039439    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"]
Oct 30 20:08:49.043835 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.043815    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.107778 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.107590    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"]
Oct 30 20:08:49.131241 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.131206    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm" event={"ID":"3017938e-ba45-4d82-b5bb-15919e2b413e","Type":"ContainerStarted","Data":"9dde582e812e89b47ecf1f64de36c322351fc581e5cca5855f9992b25b87c446"}
Oct 30 20:08:49.131406 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.131267    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm" event={"ID":"3017938e-ba45-4d82-b5bb-15919e2b413e","Type":"ContainerStarted","Data":"7fef5f007078a6f6c6a9478fd71c8ac293e1f42d7d4942a2be8644227227de6e"}
Oct 30 20:08:49.174238 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.174134    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.174238 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.174181    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.174511 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.174291    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.174511 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.174351    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8847b33-95a0-4136-9d77-82b9008d9d7d-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.174511 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.174430    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxfwf\" (UniqueName: \"kubernetes.io/projected/e8847b33-95a0-4136-9d77-82b9008d9d7d-kube-api-access-hxfwf\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.174511 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.174485    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.174706 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.174684    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.174742 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.174729    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.174891 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.174875    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.276601 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.276564    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.276836 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.276808    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.276960 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.276873    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.276960 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.276922    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8847b33-95a0-4136-9d77-82b9008d9d7d-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.277071 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.276964    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxfwf\" (UniqueName: \"kubernetes.io/projected/e8847b33-95a0-4136-9d77-82b9008d9d7d-kube-api-access-hxfwf\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.277071 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.277002    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.277172 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.277147    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.277220 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.277182    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.277344 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.277324    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.278134 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.278109    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-credential-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.278599 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.278574    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-workload-certs\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.278915 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.278888    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-data\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.280499 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.279645    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8847b33-95a0-4136-9d77-82b9008d9d7d-istiod-ca-cert\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.280499 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.279649    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-workload-socket\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.283146 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.283121    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-podinfo\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.283870 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.283851    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-envoy\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.302675 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.302624    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm" podStartSLOduration=2.302607798 podStartE2EDuration="2.302607798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:49.300426456 +0000 UTC m=+1568.032641846" watchObservedRunningTime="2026-10-30 20:08:49.302607798 +0000 UTC m=+1568.034823189"
Oct 30 20:08:49.352545 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.352507    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-token\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.352897 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.352617    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxfwf\" (UniqueName: \"kubernetes.io/projected/e8847b33-95a0-4136-9d77-82b9008d9d7d-kube-api-access-hxfwf\") pod \"e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") " pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.360443 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.360382    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:49.376221 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.376198    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:49.377958 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.377930    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 20:08:49.378079 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.377988    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm" podUID="3017938e-ba45-4d82-b5bb-15919e2b413e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 20:08:49.831099 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.831070    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"]
Oct 30 20:08:49.832230 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:08:49.832200    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8847b33_95a0_4136_9d77_82b9008d9d7d.slice/crio-047651553b45e24af711071a2cb8901b47bd4814fe3808828c3dce1ec5a5e499 WatchSource:0}: Error finding container 047651553b45e24af711071a2cb8901b47bd4814fe3808828c3dce1ec5a5e499: Status 404 returned error can't find the container with id 047651553b45e24af711071a2cb8901b47bd4814fe3808828c3dce1ec5a5e499
Oct 30 20:08:49.834738 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.834707    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:08:49.834812 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.834771    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:08:49.834812 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:49.834801    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:08:50.164576 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:50.164481    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn" event={"ID":"e8847b33-95a0-4136-9d77-82b9008d9d7d","Type":"ContainerStarted","Data":"53dbae9d1c4b4e7e6b70cf30f5a02413cbbed177020c682b402883e3a232be70"}
Oct 30 20:08:50.164576 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:50.164541    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn" event={"ID":"e8847b33-95a0-4136-9d77-82b9008d9d7d","Type":"ContainerStarted","Data":"047651553b45e24af711071a2cb8901b47bd4814fe3808828c3dce1ec5a5e499"}
Oct 30 20:08:50.288360 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:50.288303    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn" podStartSLOduration=2.28828872 podStartE2EDuration="2.28828872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:50.285618709 +0000 UTC m=+1569.017834111" watchObservedRunningTime="2026-10-30 20:08:50.28828872 +0000 UTC m=+1569.020504109"
Oct 30 20:08:50.363794 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:50.363313    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:50.363794 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:50.363475    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:08:50.363794 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:50.363523    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn" podUID="e8847b33-95a0-4136-9d77-82b9008d9d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 20:08:50.375949 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:50.375915    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 20:08:50.376076 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:50.375972    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm" podUID="3017938e-ba45-4d82-b5bb-15919e2b413e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 20:08:51.361477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:51.360856    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:08:51.361477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:51.360922    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn" podUID="e8847b33-95a0-4136-9d77-82b9008d9d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 20:08:51.383490 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:51.382886    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 20:08:51.383490 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:51.382947    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm" podUID="3017938e-ba45-4d82-b5bb-15919e2b413e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 20:08:52.366477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:52.365722    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:08:52.366477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:52.365786    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn" podUID="e8847b33-95a0-4136-9d77-82b9008d9d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 20:08:52.387478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:52.385946    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 20:08:52.387478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:52.386046    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm" podUID="3017938e-ba45-4d82-b5bb-15919e2b413e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 20:08:53.362480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:53.362084    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:08:53.362480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:53.362159    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn" podUID="e8847b33-95a0-4136-9d77-82b9008d9d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 20:08:53.386505 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:53.383930    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 20:08:53.386505 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:53.384017    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm" podUID="3017938e-ba45-4d82-b5bb-15919e2b413e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 20:08:53.594616 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:53.594574    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46952085-f6fjz"]
Oct 30 20:08:53.600835 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:53.600809    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46952085-f6fjz"
Oct 30 20:08:53.695520 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:53.695396    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x95p\" (UniqueName: \"kubernetes.io/projected/00217548-5622-4e86-b9f0-3ea049703cd8-kube-api-access-5x95p\") pod \"maas-api-revoke-sub-46952085-f6fjz\" (UID: \"00217548-5622-4e86-b9f0-3ea049703cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46952085-f6fjz"
Oct 30 20:08:53.695974 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:53.695951    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00217548-5622-4e86-b9f0-3ea049703cd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46952085-f6fjz\" (UID: \"00217548-5622-4e86-b9f0-3ea049703cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46952085-f6fjz"
Oct 30 20:08:53.696119 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:53.696103    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00217548-5622-4e86-b9f0-3ea049703cd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46952085-f6fjz\" (UID: \"00217548-5622-4e86-b9f0-3ea049703cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46952085-f6fjz"
Oct 30 20:08:53.696751 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:53.696724    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46952085-f6fjz"]
Oct 30 20:08:53.696923 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:53.696772    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:08:53.697035 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:53.696795    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:08:53.798971 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:53.798932    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00217548-5622-4e86-b9f0-3ea049703cd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46952085-f6fjz\" (UID: \"00217548-5622-4e86-b9f0-3ea049703cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46952085-f6fjz"
Oct 30 20:08:53.799146 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:53.799071    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x95p\" (UniqueName: \"kubernetes.io/projected/00217548-5622-4e86-b9f0-3ea049703cd8-kube-api-access-5x95p\") pod \"maas-api-revoke-sub-46952085-f6fjz\" (UID: \"00217548-5622-4e86-b9f0-3ea049703cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46952085-f6fjz"
Oct 30 20:08:53.799224 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:53.799210    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00217548-5622-4e86-b9f0-3ea049703cd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46952085-f6fjz\" (UID: \"00217548-5622-4e86-b9f0-3ea049703cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46952085-f6fjz"
Oct 30 20:08:53.800148 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:53.800115    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00217548-5622-4e86-b9f0-3ea049703cd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46952085-f6fjz\" (UID: \"00217548-5622-4e86-b9f0-3ea049703cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46952085-f6fjz"
Oct 30 20:08:53.802021 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:53.801998    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00217548-5622-4e86-b9f0-3ea049703cd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46952085-f6fjz\" (UID: \"00217548-5622-4e86-b9f0-3ea049703cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46952085-f6fjz"
Oct 30 20:08:53.887297 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:53.887245    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x95p\" (UniqueName: \"kubernetes.io/projected/00217548-5622-4e86-b9f0-3ea049703cd8-kube-api-access-5x95p\") pod \"maas-api-revoke-sub-46952085-f6fjz\" (UID: \"00217548-5622-4e86-b9f0-3ea049703cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46952085-f6fjz"
Oct 30 20:08:53.918901 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:53.918857    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46952085-f6fjz"
Oct 30 20:08:54.386353 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:54.385598    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 20:08:54.386353 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:54.385668    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm" podUID="3017938e-ba45-4d82-b5bb-15919e2b413e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 20:08:54.386353 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:54.386121    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:08:54.386353 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:54.386162    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn" podUID="e8847b33-95a0-4136-9d77-82b9008d9d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 20:08:54.394531 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:54.394499    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46952085-f6fjz"]
Oct 30 20:08:54.394943 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:08:54.394916    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00217548_5622_4e86_b9f0_3ea049703cd8.slice/crio-829f43d072d2dafae7828091bb50ead4f4319772dbab89d37613d646c435b582 WatchSource:0}: Error finding container 829f43d072d2dafae7828091bb50ead4f4319772dbab89d37613d646c435b582: Status 404 returned error can't find the container with id 829f43d072d2dafae7828091bb50ead4f4319772dbab89d37613d646c435b582
Oct 30 20:08:55.157477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:55.157275    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b1cf7b2-hhgt2"]
Oct 30 20:08:55.167809 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:55.167778    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1cf7b2-hhgt2"
Oct 30 20:08:55.213655 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:55.213620    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b1cf7b2-hhgt2"]
Oct 30 20:08:55.223155 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:55.223114    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb6f6f8-5072-4cb4-bb24-73ce5f4f1509-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b1cf7b2-hhgt2\" (UID: \"efb6f6f8-5072-4cb4-bb24-73ce5f4f1509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1cf7b2-hhgt2"
Oct 30 20:08:55.223383 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:55.223311    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2m97\" (UniqueName: \"kubernetes.io/projected/efb6f6f8-5072-4cb4-bb24-73ce5f4f1509-kube-api-access-q2m97\") pod \"maas-api-revoke-sub-7b1cf7b2-hhgt2\" (UID: \"efb6f6f8-5072-4cb4-bb24-73ce5f4f1509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1cf7b2-hhgt2"
Oct 30 20:08:55.223383 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:55.223363    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb6f6f8-5072-4cb4-bb24-73ce5f4f1509-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b1cf7b2-hhgt2\" (UID: \"efb6f6f8-5072-4cb4-bb24-73ce5f4f1509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1cf7b2-hhgt2"
Oct 30 20:08:55.283252 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:55.283204    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46952085-f6fjz" event={"ID":"00217548-5622-4e86-b9f0-3ea049703cd8","Type":"ContainerStarted","Data":"7a39d705719cc9f7013ca49b6a4ef2cd8eb9b42f48b0c26bbb95257d46741a2d"}
Oct 30 20:08:55.283436 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:55.283280    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46952085-f6fjz" event={"ID":"00217548-5622-4e86-b9f0-3ea049703cd8","Type":"ContainerStarted","Data":"829f43d072d2dafae7828091bb50ead4f4319772dbab89d37613d646c435b582"}
Oct 30 20:08:55.325969 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:55.325930    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb6f6f8-5072-4cb4-bb24-73ce5f4f1509-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b1cf7b2-hhgt2\" (UID: \"efb6f6f8-5072-4cb4-bb24-73ce5f4f1509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1cf7b2-hhgt2"
Oct 30 20:08:55.326131 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:55.326096    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2m97\" (UniqueName: \"kubernetes.io/projected/efb6f6f8-5072-4cb4-bb24-73ce5f4f1509-kube-api-access-q2m97\") pod \"maas-api-revoke-sub-7b1cf7b2-hhgt2\" (UID: \"efb6f6f8-5072-4cb4-bb24-73ce5f4f1509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1cf7b2-hhgt2"
Oct 30 20:08:55.326218 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:55.326154    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb6f6f8-5072-4cb4-bb24-73ce5f4f1509-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b1cf7b2-hhgt2\" (UID: \"efb6f6f8-5072-4cb4-bb24-73ce5f4f1509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1cf7b2-hhgt2"
Oct 30 20:08:55.326836 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:55.326808    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb6f6f8-5072-4cb4-bb24-73ce5f4f1509-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b1cf7b2-hhgt2\" (UID: \"efb6f6f8-5072-4cb4-bb24-73ce5f4f1509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1cf7b2-hhgt2"
Oct 30 20:08:55.328640 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:55.328611    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb6f6f8-5072-4cb4-bb24-73ce5f4f1509-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b1cf7b2-hhgt2\" (UID: \"efb6f6f8-5072-4cb4-bb24-73ce5f4f1509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1cf7b2-hhgt2"
Oct 30 20:08:55.366547 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:55.366501    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:08:55.366880 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:55.366570    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn" podUID="e8847b33-95a0-4136-9d77-82b9008d9d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 20:08:55.376234 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:55.376206    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 20:08:55.376357 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:55.376250    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm" podUID="3017938e-ba45-4d82-b5bb-15919e2b413e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 20:08:55.388046 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:55.388010    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2m97\" (UniqueName: \"kubernetes.io/projected/efb6f6f8-5072-4cb4-bb24-73ce5f4f1509-kube-api-access-q2m97\") pod \"maas-api-revoke-sub-7b1cf7b2-hhgt2\" (UID: \"efb6f6f8-5072-4cb4-bb24-73ce5f4f1509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1cf7b2-hhgt2"
Oct 30 20:08:55.405202 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:55.405132    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46952085-f6fjz" podStartSLOduration=2.405111952 podStartE2EDuration="2.405111952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:55.397933542 +0000 UTC m=+1574.130148956" watchObservedRunningTime="2026-10-30 20:08:55.405111952 +0000 UTC m=+1574.137327367"
Oct 30 20:08:55.485364 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:55.485277    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1cf7b2-hhgt2"
Oct 30 20:08:55.910613 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:55.910579    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b1cf7b2-hhgt2"]
Oct 30 20:08:55.911867 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:08:55.911838    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb6f6f8_5072_4cb4_bb24_73ce5f4f1509.slice/crio-aa3fc783c263903b558aa097ca2c1713a360db344d308953d11015edcb1a047d WatchSource:0}: Error finding container aa3fc783c263903b558aa097ca2c1713a360db344d308953d11015edcb1a047d: Status 404 returned error can't find the container with id aa3fc783c263903b558aa097ca2c1713a360db344d308953d11015edcb1a047d
Oct 30 20:08:56.294352 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:56.294318    5958 generic.go:356] "Generic (PLEG): container finished" podID="00217548-5622-4e86-b9f0-3ea049703cd8" containerID="7a39d705719cc9f7013ca49b6a4ef2cd8eb9b42f48b0c26bbb95257d46741a2d" exitCode=0
Oct 30 20:08:56.294787 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:56.294429    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46952085-f6fjz" event={"ID":"00217548-5622-4e86-b9f0-3ea049703cd8","Type":"ContainerDied","Data":"7a39d705719cc9f7013ca49b6a4ef2cd8eb9b42f48b0c26bbb95257d46741a2d"}
Oct 30 20:08:56.296423 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:56.296394    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1cf7b2-hhgt2" event={"ID":"efb6f6f8-5072-4cb4-bb24-73ce5f4f1509","Type":"ContainerStarted","Data":"aa3fc783c263903b558aa097ca2c1713a360db344d308953d11015edcb1a047d"}
Oct 30 20:08:56.361112 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:56.361076    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:08:56.361273 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:56.361139    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn" podUID="e8847b33-95a0-4136-9d77-82b9008d9d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 20:08:56.482755 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:56.482722    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:56.483288 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:56.483164    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:56.484225 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:56.484205    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:08:57.306236 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:57.306133    5958 generic.go:356] "Generic (PLEG): container finished" podID="efb6f6f8-5072-4cb4-bb24-73ce5f4f1509" containerID="c872f87e1bdd33996116effcca380a1c61728157428749dd669172593573f3ba" exitCode=0
Oct 30 20:08:57.306236 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:57.306219    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1cf7b2-hhgt2" event={"ID":"efb6f6f8-5072-4cb4-bb24-73ce5f4f1509","Type":"ContainerDied","Data":"c872f87e1bdd33996116effcca380a1c61728157428749dd669172593573f3ba"}
Oct 30 20:08:57.365376 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:57.365348    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:57.365713 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:57.365667    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:57.366962 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:57.366939    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:08:57.494605 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:57.494575    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46952085-f6fjz"
Oct 30 20:08:57.564280 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:57.564253    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00217548-5622-4e86-b9f0-3ea049703cd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00217548-5622-4e86-b9f0-3ea049703cd8-maas-api-xks-ca\") pod \"00217548-5622-4e86-b9f0-3ea049703cd8\" (UID: \"00217548-5622-4e86-b9f0-3ea049703cd8\") "
Oct 30 20:08:57.564419 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:57.564330    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00217548-5622-4e86-b9f0-3ea049703cd8-kube-api-access-5x95p\" (UniqueName: \"kubernetes.io/projected/00217548-5622-4e86-b9f0-3ea049703cd8-kube-api-access-5x95p\") pod \"00217548-5622-4e86-b9f0-3ea049703cd8\" (UID: \"00217548-5622-4e86-b9f0-3ea049703cd8\") "
Oct 30 20:08:57.564419 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:57.564374    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00217548-5622-4e86-b9f0-3ea049703cd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00217548-5622-4e86-b9f0-3ea049703cd8-maas-api-openshift-ca\") pod \"00217548-5622-4e86-b9f0-3ea049703cd8\" (UID: \"00217548-5622-4e86-b9f0-3ea049703cd8\") "
Oct 30 20:08:57.564883 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:57.564852    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00217548-5622-4e86-b9f0-3ea049703cd8-maas-api-openshift-ca" pod "00217548-5622-4e86-b9f0-3ea049703cd8" (UID: "00217548-5622-4e86-b9f0-3ea049703cd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:08:57.565240 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:57.565220    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00217548-5622-4e86-b9f0-3ea049703cd8-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:57.567041 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:57.567011    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00217548-5622-4e86-b9f0-3ea049703cd8-kube-api-access-5x95p" pod "00217548-5622-4e86-b9f0-3ea049703cd8" (UID: "00217548-5622-4e86-b9f0-3ea049703cd8"). InnerVolumeSpecName "kube-api-access-5x95p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:57.567132 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:57.567094    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00217548-5622-4e86-b9f0-3ea049703cd8-maas-api-xks-ca" pod "00217548-5622-4e86-b9f0-3ea049703cd8" (UID: "00217548-5622-4e86-b9f0-3ea049703cd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:08:57.666650 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:57.666598    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00217548-5622-4e86-b9f0-3ea049703cd8-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:57.666650 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:57.666642    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x95p\" (UniqueName: \"kubernetes.io/projected/00217548-5622-4e86-b9f0-3ea049703cd8-kube-api-access-5x95p\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:58.314746 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:58.314712    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46952085-f6fjz"
Oct 30 20:08:58.315131 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:58.314740    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46952085-f6fjz" event={"ID":"00217548-5622-4e86-b9f0-3ea049703cd8","Type":"ContainerDied","Data":"829f43d072d2dafae7828091bb50ead4f4319772dbab89d37613d646c435b582"}
Oct 30 20:08:58.315131 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:58.314795    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="829f43d072d2dafae7828091bb50ead4f4319772dbab89d37613d646c435b582"
Oct 30 20:08:58.474677 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:58.474652    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1cf7b2-hhgt2"
Oct 30 20:08:58.582308 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:58.582228    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efb6f6f8-5072-4cb4-bb24-73ce5f4f1509-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb6f6f8-5072-4cb4-bb24-73ce5f4f1509-maas-api-openshift-ca\") pod \"efb6f6f8-5072-4cb4-bb24-73ce5f4f1509\" (UID: \"efb6f6f8-5072-4cb4-bb24-73ce5f4f1509\") "
Oct 30 20:08:58.582308 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:58.582279    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efb6f6f8-5072-4cb4-bb24-73ce5f4f1509-kube-api-access-q2m97\" (UniqueName: \"kubernetes.io/projected/efb6f6f8-5072-4cb4-bb24-73ce5f4f1509-kube-api-access-q2m97\") pod \"efb6f6f8-5072-4cb4-bb24-73ce5f4f1509\" (UID: \"efb6f6f8-5072-4cb4-bb24-73ce5f4f1509\") "
Oct 30 20:08:58.582597 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:58.582335    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efb6f6f8-5072-4cb4-bb24-73ce5f4f1509-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb6f6f8-5072-4cb4-bb24-73ce5f4f1509-maas-api-xks-ca\") pod \"efb6f6f8-5072-4cb4-bb24-73ce5f4f1509\" (UID: \"efb6f6f8-5072-4cb4-bb24-73ce5f4f1509\") "
Oct 30 20:08:58.582682 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:58.582660    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb6f6f8-5072-4cb4-bb24-73ce5f4f1509-maas-api-openshift-ca" pod "efb6f6f8-5072-4cb4-bb24-73ce5f4f1509" (UID: "efb6f6f8-5072-4cb4-bb24-73ce5f4f1509"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:08:58.583113 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:58.583094    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb6f6f8-5072-4cb4-bb24-73ce5f4f1509-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:58.584498 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:58.584476    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb6f6f8-5072-4cb4-bb24-73ce5f4f1509-maas-api-xks-ca" pod "efb6f6f8-5072-4cb4-bb24-73ce5f4f1509" (UID: "efb6f6f8-5072-4cb4-bb24-73ce5f4f1509"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:08:58.584607 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:58.584573    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb6f6f8-5072-4cb4-bb24-73ce5f4f1509-kube-api-access-q2m97" pod "efb6f6f8-5072-4cb4-bb24-73ce5f4f1509" (UID: "efb6f6f8-5072-4cb4-bb24-73ce5f4f1509"). InnerVolumeSpecName "kube-api-access-q2m97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:58.684385 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:58.684343    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2m97\" (UniqueName: \"kubernetes.io/projected/efb6f6f8-5072-4cb4-bb24-73ce5f4f1509-kube-api-access-q2m97\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:58.684385 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:58.684378    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb6f6f8-5072-4cb4-bb24-73ce5f4f1509-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:59.324848 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:59.324808    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1cf7b2-hhgt2" event={"ID":"efb6f6f8-5072-4cb4-bb24-73ce5f4f1509","Type":"ContainerDied","Data":"aa3fc783c263903b558aa097ca2c1713a360db344d308953d11015edcb1a047d"}
Oct 30 20:08:59.324848 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:59.324834    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1cf7b2-hhgt2"
Oct 30 20:08:59.325309 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:08:59.324862    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa3fc783c263903b558aa097ca2c1713a360db344d308953d11015edcb1a047d"
Oct 30 20:09:06.025334 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:06.025300    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5be6971c-dgxhl"]
Oct 30 20:09:06.026028 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:06.026013    5958 cpu_manager.go:414] "containerMap: removing container" podUID="efb6f6f8-5072-4cb4-bb24-73ce5f4f1509" containerName="revoke-keys"
Oct 30 20:09:06.026075 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:06.026029    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb6f6f8-5072-4cb4-bb24-73ce5f4f1509" containerName="revoke-keys"
Oct 30 20:09:06.026075 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:06.026065    5958 cpu_manager.go:414] "containerMap: removing container" podUID="00217548-5622-4e86-b9f0-3ea049703cd8" containerName="revoke-keys"
Oct 30 20:09:06.026075 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:06.026071    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00217548-5622-4e86-b9f0-3ea049703cd8" containerName="revoke-keys"
Oct 30 20:09:06.026183 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:06.026169    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="00217548-5622-4e86-b9f0-3ea049703cd8" containerName="revoke-keys"
Oct 30 20:09:06.026213 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:06.026184    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb6f6f8-5072-4cb4-bb24-73ce5f4f1509" containerName="revoke-keys"
Oct 30 20:09:06.029802 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:06.029783    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be6971c-dgxhl"
Oct 30 20:09:06.083149 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:06.083129    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:09:06.133132 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:06.133099    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:09:06.137242 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:06.137205    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5be6971c-dgxhl"]
Oct 30 20:09:06.193671 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:06.193636    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ms76\" (UniqueName: \"kubernetes.io/projected/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd-kube-api-access-4ms76\") pod \"maas-api-revoke-sub-5be6971c-dgxhl\" (UID: \"1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be6971c-dgxhl"
Oct 30 20:09:06.193792 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:06.193763    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5be6971c-dgxhl\" (UID: \"1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be6971c-dgxhl"
Oct 30 20:09:06.193855 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:06.193839    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5be6971c-dgxhl\" (UID: \"1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be6971c-dgxhl"
Oct 30 20:09:06.295246 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:06.295182    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ms76\" (UniqueName: \"kubernetes.io/projected/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd-kube-api-access-4ms76\") pod \"maas-api-revoke-sub-5be6971c-dgxhl\" (UID: \"1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be6971c-dgxhl"
Oct 30 20:09:06.295389 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:06.295254    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5be6971c-dgxhl\" (UID: \"1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be6971c-dgxhl"
Oct 30 20:09:06.295389 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:06.295310    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5be6971c-dgxhl\" (UID: \"1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be6971c-dgxhl"
Oct 30 20:09:06.295896 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:06.295877    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5be6971c-dgxhl\" (UID: \"1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be6971c-dgxhl"
Oct 30 20:09:06.297947 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:06.297922    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5be6971c-dgxhl\" (UID: \"1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be6971c-dgxhl"
Oct 30 20:09:06.356658 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:06.356631    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ms76\" (UniqueName: \"kubernetes.io/projected/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd-kube-api-access-4ms76\") pod \"maas-api-revoke-sub-5be6971c-dgxhl\" (UID: \"1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be6971c-dgxhl"
Oct 30 20:09:06.611821 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:09:06.611787    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:09:06.643730 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:06.643705    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be6971c-dgxhl"
Oct 30 20:09:07.024775 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:07.024745    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5be6971c-dgxhl"]
Oct 30 20:09:07.025068 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:09:07.025046    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eacc5c0_ca6a_4a8e_b336_a1d48627e3dd.slice/crio-3c91a3af87edf1d93a0458840a56a74d8aeddf89e2142e209be88e927aec5680 WatchSource:0}: Error finding container 3c91a3af87edf1d93a0458840a56a74d8aeddf89e2142e209be88e927aec5680: Status 404 returned error can't find the container with id 3c91a3af87edf1d93a0458840a56a74d8aeddf89e2142e209be88e927aec5680
Oct 30 20:09:07.381367 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:07.381329    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be6971c-dgxhl" event={"ID":"1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd","Type":"ContainerStarted","Data":"3c91a3af87edf1d93a0458840a56a74d8aeddf89e2142e209be88e927aec5680"}
Oct 30 20:09:08.393771 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:08.393725    5958 generic.go:356] "Generic (PLEG): container finished" podID="1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd" containerID="11c794ea4aef0fe76a208212c9b8aae4ddb9a4cae0598f64eea2722335209d29" exitCode=0
Oct 30 20:09:08.394173 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:08.393795    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be6971c-dgxhl" event={"ID":"1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd","Type":"ContainerDied","Data":"11c794ea4aef0fe76a208212c9b8aae4ddb9a4cae0598f64eea2722335209d29"}
Oct 30 20:09:08.806525 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:08.805611    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9143-df1b06"]
Oct 30 20:09:08.837887 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:08.816133    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9143-df1b06"
Oct 30 20:09:08.853474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:08.848797    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v68c\" (UniqueName: \"kubernetes.io/projected/64ce64b2-3543-4572-98ab-059aff84dbec-kube-api-access-8v68c\") pod \"test-curl-9143-df1b06\" (UID: \"64ce64b2-3543-4572-98ab-059aff84dbec\") " pod="opendatahub/test-curl-9143-df1b06"
Oct 30 20:09:08.883474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:08.870832    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nts8c\""
Oct 30 20:09:08.998515 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:08.998045    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9143-df1b06"]
Oct 30 20:09:09.157490 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:09.152492    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v68c\" (UniqueName: \"kubernetes.io/projected/64ce64b2-3543-4572-98ab-059aff84dbec-kube-api-access-8v68c\") pod \"test-curl-9143-df1b06\" (UID: \"64ce64b2-3543-4572-98ab-059aff84dbec\") " pod="opendatahub/test-curl-9143-df1b06"
Oct 30 20:09:09.439007 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:09.396258    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v68c\" (UniqueName: \"kubernetes.io/projected/64ce64b2-3543-4572-98ab-059aff84dbec-kube-api-access-8v68c\") pod \"test-curl-9143-df1b06\" (UID: \"64ce64b2-3543-4572-98ab-059aff84dbec\") " pod="opendatahub/test-curl-9143-df1b06"
Oct 30 20:09:09.452473 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:09.452294    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9143-df1b06"
Oct 30 20:09:10.004322 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:10.004271    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 30 20:09:10.168328 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:10.168023    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:10.180431 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:10.179570    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 20:09:10.243863 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:10.242888    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:10.243863 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:10.243288    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:10.243863 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:10.243580    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 20:09:10.347939 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:10.347178    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:10.455655 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:10.455540    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 20:09:10.681122 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:10.681078    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:10.767707 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:10.748567    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 30 20:09:10.767707 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:10.766315    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 30 20:09:10.849875 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:10.832085    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf" podUID="a1551c50-4d89-4227-838c-4644c41de49b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:10.849875 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:10.832952    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" podUID="6e20c97b-66ce-485b-ba8d-90c2d9e1fa76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:11.032534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:11.007630    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:11.032534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:11.016689    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:09:11.417736 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:11.417689    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Oct 30 20:09:11.506645 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:11.446777    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:11.507353 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:11.456773    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:11.507353 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:11.462734    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:11.507760 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:11.462783    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 20:09:11.507953 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:11.462807    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 20:09:11.508130 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:11.470919    5958 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:09:11.729184 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:11.729086    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be6971c-dgxhl"
Oct 30 20:09:11.754172 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:11.754125    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:11.786877 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:11.757887    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd-kube-api-access-4ms76\" (UniqueName: \"kubernetes.io/projected/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd-kube-api-access-4ms76\") pod \"1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd\" (UID: \"1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd\") "
Oct 30 20:09:11.786877 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:11.757979    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd-maas-api-openshift-ca\") pod \"1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd\" (UID: \"1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd\") "
Oct 30 20:09:11.786877 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:11.758338    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd-maas-api-xks-ca\") pod \"1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd\" (UID: \"1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd\") "
Oct 30 20:09:12.068563 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.058721    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 30 20:09:12.168715 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.166392    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" podUID="6e20c97b-66ce-485b-ba8d-90c2d9e1fa76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded"
Oct 30 20:09:12.172060 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.172033    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd-maas-api-openshift-ca" pod "1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd" (UID: "1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:09:12.175562 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.174810    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd-maas-api-openshift-ca\") pod \"1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd\" (UID: \"1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd\") "
Oct 30 20:09:12.177206 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:09:12.176387    5958 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd/volumes/kubernetes.io~configmap/maas-api-openshift-ca
Oct 30 20:09:12.177206 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.176407    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd-maas-api-openshift-ca" pod "1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd" (UID: "1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:09:12.180368 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.180342    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd-kube-api-access-4ms76" pod "1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd" (UID: "1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd"). InnerVolumeSpecName "kube-api-access-4ms76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:09:12.188054 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.188028    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd-maas-api-xks-ca" pod "1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd" (UID: "1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:09:12.197527 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.193669    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9143-df1b06"]
Oct 30 20:09:12.197741 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.197721    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 20:09:12.197913 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.197899    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:09:12.198045 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.198033    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 20:09:12.198166 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.198155    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 20:09:12.198284 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.198272    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:09:12.198419 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.198407    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:09:12.213559 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.213535    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:09:12.213743 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.213728    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 20:09:12.213867 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.213855    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:09:12.285507 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.279093    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:09:12.305522 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.284091    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 20:09:12.307737 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.307207    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ms76\" (UniqueName: \"kubernetes.io/projected/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd-kube-api-access-4ms76\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:09:12.307737 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.307245    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:09:12.308843 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.308550    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 20:09:12.327904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.315128    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9143-df1b06" event={"ID":"64ce64b2-3543-4572-98ab-059aff84dbec","Type":"ContainerStarted","Data":"bd568086b1c72fbc33d443dedb6de5d054787c4a2eee41d9a95410c4c91402cb"}
Oct 30 20:09:12.327904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.318117    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be6971c-dgxhl" event={"ID":"1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd","Type":"ContainerDied","Data":"3c91a3af87edf1d93a0458840a56a74d8aeddf89e2142e209be88e927aec5680"}
Oct 30 20:09:12.327904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.318183    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c91a3af87edf1d93a0458840a56a74d8aeddf89e2142e209be88e927aec5680"
Oct 30 20:09:12.327904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.318262    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be6971c-dgxhl"
Oct 30 20:09:12.389472 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.386608    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:09:12.428964 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.428932    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 20:09:12.563779 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.563748    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:09:12.569622 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.569592    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:09:12.576883 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.576859    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 20:09:12.583531 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:12.577691    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:09:13.329519 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:13.329443    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9143-df1b06" event={"ID":"64ce64b2-3543-4572-98ab-059aff84dbec","Type":"ContainerStarted","Data":"97573bbd828db3f593013fd46fbe2caa4645d0fda44411d8276886692cac5eaa"}
Oct 30 20:09:13.871098 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:13.871034    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9143-df1b06" podStartSLOduration=4.478988025 podStartE2EDuration="5.871019314s" totalImagesPullingTime="1.392031289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:13.868245912 +0000 UTC m=+1592.600461304" watchObservedRunningTime="2026-10-30 20:09:13.871019314 +0000 UTC m=+1592.603234702"
Oct 30 20:09:15.636768 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:15.636669    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9143-df1b06"]
Oct 30 20:09:15.637729 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:15.637695    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9143-df1b06" podUID="64ce64b2-3543-4572-98ab-059aff84dbec" containerName="test-curl-9143-df1b06" containerID="cri-o://97573bbd828db3f593013fd46fbe2caa4645d0fda44411d8276886692cac5eaa" gracePeriod=2
Oct 30 20:09:15.641807 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:15.641780    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9143-df1b06"]
Oct 30 20:09:15.689079 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:09:15.689035    5958 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9143-df1b06\" is forbidden: User \"system:node:ip-10-0-142-240.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-240.us-west-1.compute.internal' and this object" podUID="64ce64b2-3543-4572-98ab-059aff84dbec" pod="opendatahub/test-curl-9143-df1b06"
Oct 30 20:09:16.574942 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.574909    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"]
Oct 30 20:09:16.575724 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.575706    5958 cpu_manager.go:414] "containerMap: removing container" podUID="1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd" containerName="revoke-keys"
Oct 30 20:09:16.575724 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.575726    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd" containerName="revoke-keys"
Oct 30 20:09:16.575821 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.575767    5958 cpu_manager.go:414] "containerMap: removing container" podUID="64ce64b2-3543-4572-98ab-059aff84dbec" containerName="test-curl-9143-df1b06"
Oct 30 20:09:16.575821 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.575772    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ce64b2-3543-4572-98ab-059aff84dbec" containerName="test-curl-9143-df1b06"
Oct 30 20:09:16.575887 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.575875    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ce64b2-3543-4572-98ab-059aff84dbec" containerName="test-curl-9143-df1b06"
Oct 30 20:09:16.575923 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.575888    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd" containerName="revoke-keys"
Oct 30 20:09:16.579731 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.579710    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.651342 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.651278    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-27915880-gw-openshift-default-dockercfg-2rdnt\""
Oct 30 20:09:16.657309 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.657271    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"]
Oct 30 20:09:16.722661 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.722619    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-token\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.722661 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.722665    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-credential-socket\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.722918 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.722826    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbp4z\" (UniqueName: \"kubernetes.io/projected/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-kube-api-access-tbp4z\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.722918 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.722859    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-podinfo\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.723018 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.722945    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-data\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.723018 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.722993    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istiod-ca-cert\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.723233 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.723215    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-envoy\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.723306 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.723254    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-workload-socket\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.723306 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.723281    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-workload-certs\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.781472 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:09:16.780731    5958 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9143-df1b06\" is forbidden: User \"system:node:ip-10-0-142-240.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-240.us-west-1.compute.internal' and this object" podUID="64ce64b2-3543-4572-98ab-059aff84dbec" pod="opendatahub/test-curl-9143-df1b06"
Oct 30 20:09:16.828491 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.828365    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbp4z\" (UniqueName: \"kubernetes.io/projected/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-kube-api-access-tbp4z\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.828728 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.828702    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-podinfo\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.828899 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.828873    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-data\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.829048 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.829031    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istiod-ca-cert\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.829968 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.829925    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-envoy\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.830144 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.830124    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-workload-socket\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.830268 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.830252    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-workload-certs\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.830499 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.830478    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-token\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.830639 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.830622    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-credential-socket\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.831098 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.831066    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-workload-certs\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.831503 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.831482    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-workload-socket\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.831900 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.831859    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-data\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.833224 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.833187    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istiod-ca-cert\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.850666 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.850628    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-credential-socket\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.851786 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.851730    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-envoy\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.856352 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.856324    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-podinfo\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.890339 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.890305    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbp4z\" (UniqueName: \"kubernetes.io/projected/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-kube-api-access-tbp4z\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:16.897409 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:16.897384    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-token\") pod \"e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") " pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:17.195868 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.195776    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:17.547553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.547308    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"]
Oct 30 20:09:17.625276 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.625243    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"]
Oct 30 20:09:17.625690 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.625601    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.699511 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.699471    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-80e4d90e-openshift-default-dockercfg-h5wsg\""
Oct 30 20:09:17.750489 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.750352    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:09:17.750489 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.750485    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:09:17.750640 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.750532    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:09:17.754726 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.753981    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"]
Oct 30 20:09:17.792738 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.792707    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-envoy\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.792881 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.792847    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-token\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.792923 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.792886    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede036c9-86b3-4d4b-95b0-88570503f1c2-istiod-ca-cert\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.792970 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.792944    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-data\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.793053 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.793033    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-workload-certs\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.793165 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.793148    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-credential-socket\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.793223 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.793179    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz6wk\" (UniqueName: \"kubernetes.io/projected/ede036c9-86b3-4d4b-95b0-88570503f1c2-kube-api-access-tz6wk\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.793304 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.793292    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-podinfo\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.793415 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.793326    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-workload-socket\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.894749 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.894705    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-podinfo\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.894908 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.894769    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-workload-socket\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.894974 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.894918    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-envoy\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.895027 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.895002    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-token\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.895081 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.895046    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede036c9-86b3-4d4b-95b0-88570503f1c2-istiod-ca-cert\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.895137 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.895100    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-data\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.895186 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.895160    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-workload-certs\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.895290 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.895270    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-credential-socket\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.895358 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.895308    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz6wk\" (UniqueName: \"kubernetes.io/projected/ede036c9-86b3-4d4b-95b0-88570503f1c2-kube-api-access-tz6wk\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.895699 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.895671    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-workload-socket\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.900279 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.899837    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-workload-certs\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.901077 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.900544    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede036c9-86b3-4d4b-95b0-88570503f1c2-istiod-ca-cert\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.901077 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.900777    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-data\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.901632 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.901350    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-credential-socket\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.906329 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.906303    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-envoy\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.908928 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.908907    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-podinfo\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.970100 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.970066    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz6wk\" (UniqueName: \"kubernetes.io/projected/ede036c9-86b3-4d4b-95b0-88570503f1c2-kube-api-access-tz6wk\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.970543 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.970520    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-token\") pod \"e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") " pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:17.986508 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:17.985646    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9143-df1b06"
Oct 30 20:09:18.098568 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:18.098431    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ce64b2-3543-4572-98ab-059aff84dbec-kube-api-access-8v68c\" (UniqueName: \"kubernetes.io/projected/64ce64b2-3543-4572-98ab-059aff84dbec-kube-api-access-8v68c\") pod \"64ce64b2-3543-4572-98ab-059aff84dbec\" (UID: \"64ce64b2-3543-4572-98ab-059aff84dbec\") "
Oct 30 20:09:18.101128 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:18.101101    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ce64b2-3543-4572-98ab-059aff84dbec-kube-api-access-8v68c" pod "64ce64b2-3543-4572-98ab-059aff84dbec" (UID: "64ce64b2-3543-4572-98ab-059aff84dbec"). InnerVolumeSpecName "kube-api-access-8v68c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:09:18.125642 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:18.125612    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d96a36be-tz9jd"]
Oct 30 20:09:18.139800 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:18.139762    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d96a36be-tz9jd"]
Oct 30 20:09:18.202410 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:18.202368    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v68c\" (UniqueName: \"kubernetes.io/projected/64ce64b2-3543-4572-98ab-059aff84dbec-kube-api-access-8v68c\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:09:18.252772 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:18.252733    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:18.377124 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:18.377027    5958 generic.go:356] "Generic (PLEG): container finished" podID="64ce64b2-3543-4572-98ab-059aff84dbec" containerID="97573bbd828db3f593013fd46fbe2caa4645d0fda44411d8276886692cac5eaa" exitCode=137
Oct 30 20:09:18.377124 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:18.377123    5958 scope.go:122] "RemoveContainer" containerID="97573bbd828db3f593013fd46fbe2caa4645d0fda44411d8276886692cac5eaa"
Oct 30 20:09:18.382263 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:18.381101    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6" event={"ID":"33bf491a-644a-4a04-b8d5-5ca6842ff3e6","Type":"ContainerStarted","Data":"5041afe3a7604006b478cc2e63cf910ce0291f8f47d04fc298fa48ad9f16383b"}
Oct 30 20:09:18.382263 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:18.381168    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6" event={"ID":"33bf491a-644a-4a04-b8d5-5ca6842ff3e6","Type":"ContainerStarted","Data":"11d0e856d5f2f4298ecad1ff6049a94d01f6729f68e80f5683ece27871ffd498"}
Oct 30 20:09:18.395644 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:18.395596    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9143-df1b06"
Oct 30 20:09:18.402651 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:18.402624    5958 scope.go:122] "RemoveContainer" containerID="97573bbd828db3f593013fd46fbe2caa4645d0fda44411d8276886692cac5eaa"
Oct 30 20:09:18.403426 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:09:18.403057    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97573bbd828db3f593013fd46fbe2caa4645d0fda44411d8276886692cac5eaa\": container with ID starting with 97573bbd828db3f593013fd46fbe2caa4645d0fda44411d8276886692cac5eaa not found: ID does not exist" containerID="97573bbd828db3f593013fd46fbe2caa4645d0fda44411d8276886692cac5eaa"
Oct 30 20:09:18.403426 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:18.403099    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97573bbd828db3f593013fd46fbe2caa4645d0fda44411d8276886692cac5eaa"} err="failed to get container status \"97573bbd828db3f593013fd46fbe2caa4645d0fda44411d8276886692cac5eaa\": rpc error: code = NotFound desc = could not find container \"97573bbd828db3f593013fd46fbe2caa4645d0fda44411d8276886692cac5eaa\": container with ID starting with 97573bbd828db3f593013fd46fbe2caa4645d0fda44411d8276886692cac5eaa not found: ID does not exist"
Oct 30 20:09:18.584238 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:18.584181    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6" podStartSLOduration=2.5841656349999997 podStartE2EDuration="2.584165635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:18.580698454 +0000 UTC m=+1597.312913845" watchObservedRunningTime="2026-10-30 20:09:18.584165635 +0000 UTC m=+1597.316381026"
Oct 30 20:09:18.640690 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:09:18.640567    5958 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9143-df1b06\" is forbidden: User \"system:node:ip-10-0-142-240.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-240.us-west-1.compute.internal' and this object" podUID="64ce64b2-3543-4572-98ab-059aff84dbec" pod="opendatahub/test-curl-9143-df1b06"
Oct 30 20:09:18.914575 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:18.874268    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"]
Oct 30 20:09:18.915164 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:18.875550    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:09:18.915572 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:18.915526    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:09:18.915818 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:18.915786    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:09:19.196470 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:19.196380    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:19.201829 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:19.201787    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:19.253199 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:09:19.253148    5958 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9143-df1b06\" is forbidden: User \"system:node:ip-10-0-142-240.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-240.us-west-1.compute.internal' and this object" podUID="64ce64b2-3543-4572-98ab-059aff84dbec" pod="opendatahub/test-curl-9143-df1b06"
Oct 30 20:09:19.404320 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:19.404286    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ba655b7-a698-4157-91c9-5849e1b6bbd6" path="/var/lib/kubelet/pods/1ba655b7-a698-4157-91c9-5849e1b6bbd6/volumes"
Oct 30 20:09:19.404862 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:19.404838    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ce64b2-3543-4572-98ab-059aff84dbec" path="/var/lib/kubelet/pods/64ce64b2-3543-4572-98ab-059aff84dbec/volumes"
Oct 30 20:09:19.405220 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:19.405204    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:19.405283 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:19.405246    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c" event={"ID":"ede036c9-86b3-4d4b-95b0-88570503f1c2","Type":"ContainerStarted","Data":"f5f7b43196475efd0db9db8770deabe65b8a5276355d90df18320252ede3dbb2"}
Oct 30 20:09:19.405336 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:19.405298    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:09:19.405336 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:19.405314    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c" event={"ID":"ede036c9-86b3-4d4b-95b0-88570503f1c2","Type":"ContainerStarted","Data":"06324203575c67e784afc8e85b4466612cec1983e898e9579eb8fb40eedde7a0"}
Oct 30 20:09:19.444986 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:09:19.444930    5958 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9143-df1b06\" is forbidden: User \"system:node:ip-10-0-142-240.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-240.us-west-1.compute.internal' and this object" podUID="64ce64b2-3543-4572-98ab-059aff84dbec" pod="opendatahub/test-curl-9143-df1b06"
Oct 30 20:09:19.586498 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:19.586425    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c" podStartSLOduration=2.586410534 podStartE2EDuration="2.586410534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:19.582244867 +0000 UTC m=+1598.314460280" watchObservedRunningTime="2026-10-30 20:09:19.586410534 +0000 UTC m=+1598.318625923"
Oct 30 20:09:20.253175 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:20.253139    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:20.258958 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:20.258927    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:20.403047 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:20.403009    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:20.403996 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:20.403977    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:09:24.552687 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:24.552649    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn"]
Oct 30 20:09:24.560592 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:24.560561    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn"
Oct 30 20:09:24.616125 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:24.616022    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn"]
Oct 30 20:09:24.661278 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:24.661246    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-27915880\""
Oct 30 20:09:24.661498 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:24.661342    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-27915880-dockercfg-vhwwb\""
Oct 30 20:09:24.669371 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:24.669334    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvs2m\" (UniqueName: \"kubernetes.io/projected/6530268f-b6ee-4315-ab68-affa9a4057f9-kube-api-access-cvs2m\") pod \"payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn\" (UID: \"6530268f-b6ee-4315-ab68-affa9a4057f9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn"
Oct 30 20:09:24.669676 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:24.669648    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6530268f-b6ee-4315-ab68-affa9a4057f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn\" (UID: \"6530268f-b6ee-4315-ab68-affa9a4057f9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn"
Oct 30 20:09:24.747141 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:24.747102    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9"]
Oct 30 20:09:24.751695 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:24.751674    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9"
Oct 30 20:09:24.771141 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:24.771110    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6530268f-b6ee-4315-ab68-affa9a4057f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn\" (UID: \"6530268f-b6ee-4315-ab68-affa9a4057f9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn"
Oct 30 20:09:24.771325 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:24.771283    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvs2m\" (UniqueName: \"kubernetes.io/projected/6530268f-b6ee-4315-ab68-affa9a4057f9-kube-api-access-cvs2m\") pod \"payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn\" (UID: \"6530268f-b6ee-4315-ab68-affa9a4057f9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn"
Oct 30 20:09:24.771838 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:24.771807    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6530268f-b6ee-4315-ab68-affa9a4057f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn\" (UID: \"6530268f-b6ee-4315-ab68-affa9a4057f9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn"
Oct 30 20:09:24.822063 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:24.822025    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9"]
Oct 30 20:09:24.838634 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:24.838564    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvs2m\" (UniqueName: \"kubernetes.io/projected/6530268f-b6ee-4315-ab68-affa9a4057f9-kube-api-access-cvs2m\") pod \"payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn\" (UID: \"6530268f-b6ee-4315-ab68-affa9a4057f9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn"
Oct 30 20:09:24.874936 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:24.874893    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fxpn\" (UniqueName: \"kubernetes.io/projected/ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4-kube-api-access-2fxpn\") pod \"payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9\" (UID: \"ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4\") " pod="openshift-ingress/payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9"
Oct 30 20:09:24.875133 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:24.875111    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9\" (UID: \"ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4\") " pod="openshift-ingress/payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9"
Oct 30 20:09:24.896729 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:24.890766    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn"
Oct 30 20:09:24.983527 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:24.983492    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fxpn\" (UniqueName: \"kubernetes.io/projected/ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4-kube-api-access-2fxpn\") pod \"payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9\" (UID: \"ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4\") " pod="openshift-ingress/payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9"
Oct 30 20:09:24.983955 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:24.983927    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9\" (UID: \"ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4\") " pod="openshift-ingress/payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9"
Oct 30 20:09:24.984826 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:24.984795    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9\" (UID: \"ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4\") " pod="openshift-ingress/payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9"
Oct 30 20:09:25.048533 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:25.048477    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fxpn\" (UniqueName: \"kubernetes.io/projected/ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4-kube-api-access-2fxpn\") pod \"payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9\" (UID: \"ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4\") " pod="openshift-ingress/payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9"
Oct 30 20:09:25.071555 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:25.070290    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9"
Oct 30 20:09:25.375437 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:09:25.375407    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6530268f_b6ee_4315_ab68_affa9a4057f9.slice/crio-20083a301c6852bf5a34302351a01607c8410559f28dc4b59dfe23d0c87fa799 WatchSource:0}: Error finding container 20083a301c6852bf5a34302351a01607c8410559f28dc4b59dfe23d0c87fa799: Status 404 returned error can't find the container with id 20083a301c6852bf5a34302351a01607c8410559f28dc4b59dfe23d0c87fa799
Oct 30 20:09:25.376822 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:25.376791    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn"]
Oct 30 20:09:25.445127 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:25.445101    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn" event={"ID":"6530268f-b6ee-4315-ab68-affa9a4057f9","Type":"ContainerStarted","Data":"20083a301c6852bf5a34302351a01607c8410559f28dc4b59dfe23d0c87fa799"}
Oct 30 20:09:25.478815 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:25.478784    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9"]
Oct 30 20:09:26.456152 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:26.456080    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9" event={"ID":"ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4","Type":"ContainerStarted","Data":"b2a191afe3a6b2c43f49fad6a86dfd13449610e684e2873038bb4f106f462c06"}
Oct 30 20:09:26.456728 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:26.456179    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9" event={"ID":"ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4","Type":"ContainerStarted","Data":"a7f0ddfc832ace5106e258b370552c99041df14aa5df463b6cdf4c4624dda9d8"}
Oct 30 20:09:26.456728 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:26.456217    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9"
Oct 30 20:09:26.458794 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:26.458767    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn" event={"ID":"6530268f-b6ee-4315-ab68-affa9a4057f9","Type":"ContainerStarted","Data":"68bb14775d4f275a5dc211dbc3c4a1da55b69ce02502044281ae7fee4de11236"}
Oct 30 20:09:26.459010 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:26.458983    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn"
Oct 30 20:09:26.580641 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:26.580582    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9" podStartSLOduration=2.580564599 podStartE2EDuration="2.580564599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:26.573498613 +0000 UTC m=+1605.305714007" watchObservedRunningTime="2026-10-30 20:09:26.580564599 +0000 UTC m=+1605.312779993"
Oct 30 20:09:26.697266 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:26.697205    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn" podStartSLOduration=2.697189137 podStartE2EDuration="2.697189137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:26.693795152 +0000 UTC m=+1605.426010542" watchObservedRunningTime="2026-10-30 20:09:26.697189137 +0000 UTC m=+1605.429404526"
Oct 30 20:09:27.433151 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:27.433120    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc"]
Oct 30 20:09:27.438059 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:27.438037    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc"
Oct 30 20:09:27.494626 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:27.494592    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-80e4d90e\""
Oct 30 20:09:27.522753 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:27.522714    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc"]
Oct 30 20:09:27.533653 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:27.533622    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8j84\" (UniqueName: \"kubernetes.io/projected/fc1532ea-e75f-4f6e-9791-8e95bcf51cf8-kube-api-access-j8j84\") pod \"payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc\" (UID: \"fc1532ea-e75f-4f6e-9791-8e95bcf51cf8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc"
Oct 30 20:09:27.533934 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:27.533909    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1532ea-e75f-4f6e-9791-8e95bcf51cf8-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc\" (UID: \"fc1532ea-e75f-4f6e-9791-8e95bcf51cf8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc"
Oct 30 20:09:27.543353 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:27.543305    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-80e4d90e-dockercfg-kjjs6\""
Oct 30 20:09:27.576010 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:27.575974    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv"]
Oct 30 20:09:27.580772 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:27.580746    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv"
Oct 30 20:09:27.641467 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:27.641414    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1532ea-e75f-4f6e-9791-8e95bcf51cf8-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc\" (UID: \"fc1532ea-e75f-4f6e-9791-8e95bcf51cf8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc"
Oct 30 20:09:27.641636 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:27.641532    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br52p\" (UniqueName: \"kubernetes.io/projected/026dbd11-0bd0-445d-8e44-388ceaf9418a-kube-api-access-br52p\") pod \"payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv\" (UID: \"026dbd11-0bd0-445d-8e44-388ceaf9418a\") " pod="openshift-ingress/payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv"
Oct 30 20:09:27.641761 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:27.641746    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/026dbd11-0bd0-445d-8e44-388ceaf9418a-plugins-config-volume\") pod \"payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv\" (UID: \"026dbd11-0bd0-445d-8e44-388ceaf9418a\") " pod="openshift-ingress/payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv"
Oct 30 20:09:27.641795 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:27.641783    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8j84\" (UniqueName: \"kubernetes.io/projected/fc1532ea-e75f-4f6e-9791-8e95bcf51cf8-kube-api-access-j8j84\") pod \"payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc\" (UID: \"fc1532ea-e75f-4f6e-9791-8e95bcf51cf8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc"
Oct 30 20:09:27.643524 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:27.643126    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1532ea-e75f-4f6e-9791-8e95bcf51cf8-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc\" (UID: \"fc1532ea-e75f-4f6e-9791-8e95bcf51cf8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc"
Oct 30 20:09:27.656371 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:27.656339    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv"]
Oct 30 20:09:27.711525 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:27.707601    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8j84\" (UniqueName: \"kubernetes.io/projected/fc1532ea-e75f-4f6e-9791-8e95bcf51cf8-kube-api-access-j8j84\") pod \"payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc\" (UID: \"fc1532ea-e75f-4f6e-9791-8e95bcf51cf8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc"
Oct 30 20:09:27.748574 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:27.744254    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/026dbd11-0bd0-445d-8e44-388ceaf9418a-plugins-config-volume\") pod \"payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv\" (UID: \"026dbd11-0bd0-445d-8e44-388ceaf9418a\") " pod="openshift-ingress/payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv"
Oct 30 20:09:27.748574 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:27.744545    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br52p\" (UniqueName: \"kubernetes.io/projected/026dbd11-0bd0-445d-8e44-388ceaf9418a-kube-api-access-br52p\") pod \"payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv\" (UID: \"026dbd11-0bd0-445d-8e44-388ceaf9418a\") " pod="openshift-ingress/payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv"
Oct 30 20:09:27.748574 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:27.745849    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/026dbd11-0bd0-445d-8e44-388ceaf9418a-plugins-config-volume\") pod \"payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv\" (UID: \"026dbd11-0bd0-445d-8e44-388ceaf9418a\") " pod="openshift-ingress/payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv"
Oct 30 20:09:27.753304 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:27.753268    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc"
Oct 30 20:09:27.807186 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:27.807150    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br52p\" (UniqueName: \"kubernetes.io/projected/026dbd11-0bd0-445d-8e44-388ceaf9418a-kube-api-access-br52p\") pod \"payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv\" (UID: \"026dbd11-0bd0-445d-8e44-388ceaf9418a\") " pod="openshift-ingress/payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv"
Oct 30 20:09:27.923799 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:27.923663    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv"
Oct 30 20:09:28.156475 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:09:28.156103    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1532ea_e75f_4f6e_9791_8e95bcf51cf8.slice/crio-837e6705e86a8727fa86c14a0c16bd02c9eb46ae02599338b7b1494524eb4d23 WatchSource:0}: Error finding container 837e6705e86a8727fa86c14a0c16bd02c9eb46ae02599338b7b1494524eb4d23: Status 404 returned error can't find the container with id 837e6705e86a8727fa86c14a0c16bd02c9eb46ae02599338b7b1494524eb4d23
Oct 30 20:09:28.156475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:28.156383    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc"]
Oct 30 20:09:28.360734 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:09:28.360703    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod026dbd11_0bd0_445d_8e44_388ceaf9418a.slice/crio-a343a5c04c2a71d3708989ea9905f64ea75c718c982a5c73704a42743ecd87de WatchSource:0}: Error finding container a343a5c04c2a71d3708989ea9905f64ea75c718c982a5c73704a42743ecd87de: Status 404 returned error can't find the container with id a343a5c04c2a71d3708989ea9905f64ea75c718c982a5c73704a42743ecd87de
Oct 30 20:09:28.361781 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:28.361756    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv"]
Oct 30 20:09:28.476010 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:28.475970    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc" event={"ID":"fc1532ea-e75f-4f6e-9791-8e95bcf51cf8","Type":"ContainerStarted","Data":"2c5caeafd700ae0de212bbc5e4bf1fae2b125a4175e26e3621378f69525e49f8"}
Oct 30 20:09:28.476135 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:28.476034    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc" event={"ID":"fc1532ea-e75f-4f6e-9791-8e95bcf51cf8","Type":"ContainerStarted","Data":"837e6705e86a8727fa86c14a0c16bd02c9eb46ae02599338b7b1494524eb4d23"}
Oct 30 20:09:28.476220 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:28.476200    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc"
Oct 30 20:09:28.477738 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:28.477713    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv" event={"ID":"026dbd11-0bd0-445d-8e44-388ceaf9418a","Type":"ContainerStarted","Data":"5c382a5097345aad32763d58eaa5fdded497870bbd9942ef8f8ebf947ef3ffa2"}
Oct 30 20:09:28.477837 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:28.477757    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv" event={"ID":"026dbd11-0bd0-445d-8e44-388ceaf9418a","Type":"ContainerStarted","Data":"a343a5c04c2a71d3708989ea9905f64ea75c718c982a5c73704a42743ecd87de"}
Oct 30 20:09:28.477837 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:28.477783    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv"
Oct 30 20:09:28.683501 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:28.683389    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc" podStartSLOduration=1.683375359 podStartE2EDuration="1.683375359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:28.680744319 +0000 UTC m=+1607.412959718" watchObservedRunningTime="2026-10-30 20:09:28.683375359 +0000 UTC m=+1607.415590748"
Oct 30 20:09:28.877982 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:28.877907    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv" podStartSLOduration=1.8778842679999999 podStartE2EDuration="1.877884268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:28.851151308 +0000 UTC m=+1607.583366697" watchObservedRunningTime="2026-10-30 20:09:28.877884268 +0000 UTC m=+1607.610099692"
Oct 30 20:09:32.053442 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:32.053401    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52d5c320-688gn"]
Oct 30 20:09:32.103540 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:32.103503    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52d5c320-688gn"]
Oct 30 20:09:33.400325 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:33.400289    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b73e696-124a-4cab-bb04-442487660026" path="/var/lib/kubelet/pods/0b73e696-124a-4cab-bb04-442487660026/volumes"
Oct 30 20:09:36.608850 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:09:36.608809    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:09:37.469650 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:37.469619    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9"
Oct 30 20:09:37.469817 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:37.469701    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn"
Oct 30 20:09:38.479042 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:38.479001    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk"]
Oct 30 20:09:38.486895 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:38.486865    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk"
Oct 30 20:09:38.544102 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:38.544064    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk"]
Oct 30 20:09:38.587275 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:38.587223    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d7ca5342\""
Oct 30 20:09:38.587633 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:38.587381    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d7ca5342-dockercfg-76czr\""
Oct 30 20:09:38.591263 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:38.591226    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr5ds\" (UniqueName: \"kubernetes.io/projected/4017f735-ee90-40c3-ba71-4e57ddf03681-kube-api-access-pr5ds\") pod \"payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk\" (UID: \"4017f735-ee90-40c3-ba71-4e57ddf03681\") " pod="openshift-ingress/payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk"
Oct 30 20:09:38.591601 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:38.591573    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4017f735-ee90-40c3-ba71-4e57ddf03681-plugins-config-volume\") pod \"payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk\" (UID: \"4017f735-ee90-40c3-ba71-4e57ddf03681\") " pod="openshift-ingress/payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk"
Oct 30 20:09:38.693424 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:38.693388    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4017f735-ee90-40c3-ba71-4e57ddf03681-plugins-config-volume\") pod \"payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk\" (UID: \"4017f735-ee90-40c3-ba71-4e57ddf03681\") " pod="openshift-ingress/payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk"
Oct 30 20:09:38.693638 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:38.693603    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr5ds\" (UniqueName: \"kubernetes.io/projected/4017f735-ee90-40c3-ba71-4e57ddf03681-kube-api-access-pr5ds\") pod \"payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk\" (UID: \"4017f735-ee90-40c3-ba71-4e57ddf03681\") " pod="openshift-ingress/payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk"
Oct 30 20:09:38.694189 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:38.694155    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4017f735-ee90-40c3-ba71-4e57ddf03681-plugins-config-volume\") pod \"payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk\" (UID: \"4017f735-ee90-40c3-ba71-4e57ddf03681\") " pod="openshift-ingress/payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk"
Oct 30 20:09:38.759012 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:38.758935    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr5ds\" (UniqueName: \"kubernetes.io/projected/4017f735-ee90-40c3-ba71-4e57ddf03681-kube-api-access-pr5ds\") pod \"payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk\" (UID: \"4017f735-ee90-40c3-ba71-4e57ddf03681\") " pod="openshift-ingress/payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk"
Oct 30 20:09:38.817003 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:38.816962    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk"
Oct 30 20:09:39.225510 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:39.225480    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk"]
Oct 30 20:09:39.226195 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:09:39.226170    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4017f735_ee90_40c3_ba71_4e57ddf03681.slice/crio-82b6925367be3f8b3dc220cc8f3fa81e1e4a3aa52046e6d2009911d19a474005 WatchSource:0}: Error finding container 82b6925367be3f8b3dc220cc8f3fa81e1e4a3aa52046e6d2009911d19a474005: Status 404 returned error can't find the container with id 82b6925367be3f8b3dc220cc8f3fa81e1e4a3aa52046e6d2009911d19a474005
Oct 30 20:09:39.490128 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:39.490049    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc"
Oct 30 20:09:39.490128 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:39.490114    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv"
Oct 30 20:09:39.588216 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:39.588180    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk" event={"ID":"4017f735-ee90-40c3-ba71-4e57ddf03681","Type":"ContainerStarted","Data":"3f7e979e0577dcc61d3169c07f1350e9c4324bcbf6f7ce524078632406a4c77b"}
Oct 30 20:09:39.588400 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:39.588234    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk" event={"ID":"4017f735-ee90-40c3-ba71-4e57ddf03681","Type":"ContainerStarted","Data":"82b6925367be3f8b3dc220cc8f3fa81e1e4a3aa52046e6d2009911d19a474005"}
Oct 30 20:09:39.588400 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:39.588273    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk"
Oct 30 20:09:48.475474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:48.474654    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:48.580523 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:48.576528    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf" podUID="a1551c50-4d89-4227-838c-4644c41de49b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:48.612477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:48.594253    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:48.635492 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:48.633426    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:48.719025 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:48.713734    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" podUID="6e20c97b-66ce-485b-ba8d-90c2d9e1fa76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:49.070475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:49.069550    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:49.273490 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:49.270575    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:49.289615 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:49.289557    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" podUID="6e20c97b-66ce-485b-ba8d-90c2d9e1fa76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:49.310657 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:49.310614    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:49.382697 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:49.382585    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:49.484603 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:49.484549    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:49.516814 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:49.514878    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:49.516814 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:49.515429    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 30 20:09:49.516814 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:49.515835    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:49.536929 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:49.536593    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 20:09:49.575600 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:49.575550    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:49.580585 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:49.580539    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf" podUID="a1551c50-4d89-4227-838c-4644c41de49b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:49.581251 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:49.581228    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:09:49.620746 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:49.618549    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 20:09:49.620746 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:49.619008    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:09:49.635149 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:49.635047    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:49.635814 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:49.635790    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 20:09:49.636993 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:49.636965    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:49.708599 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:49.704121    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk" podStartSLOduration=11.704098622 podStartE2EDuration="11.704098622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:39.854666386 +0000 UTC m=+1618.586881779" watchObservedRunningTime="2026-10-30 20:09:49.704098622 +0000 UTC m=+1628.436314014"
Oct 30 20:09:50.084084 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:50.084036    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:50.114772 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:50.114731    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:09:50.246516 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:50.165640    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:50.290058 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:50.290013    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" podUID="6e20c97b-66ce-485b-ba8d-90c2d9e1fa76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:50.295983 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:50.295949    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 20:09:50.636474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:50.340888    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 30 20:09:50.637706 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:50.637665    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:50.670476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:50.641676    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf" podUID="a1551c50-4d89-4227-838c-4644c41de49b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:50.670476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:50.569892    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:50.670476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:50.643973    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:50.756480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:50.569624    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:50.760549 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:50.569768    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:50.793478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:50.776502    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 30 20:09:50.954533 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:50.953017    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295" podUID="f55f875c-cde6-43b0-bd95-ead043376d8f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:51.012476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:50.987920    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:51.052479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:51.048980    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 20:09:51.052479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:51.049098    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:09:51.052479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:51.049160    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:09:51.052479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:51.049211    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:09:51.052479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:51.049249    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:09:51.124289 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:51.094920    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:51.125061 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:51.125019    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:51.249575 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:51.234864    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk"
Oct 30 20:09:51.297714 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:51.297672    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" podUID="6e20c97b-66ce-485b-ba8d-90c2d9e1fa76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:51.643237 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:51.643187    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:51.758992 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:51.647549    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:51.870482 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:51.845019    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf" podUID="a1551c50-4d89-4227-838c-4644c41de49b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:51.899933 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:51.871044    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:51.900417 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:51.871080    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Oct 30 20:09:51.955874 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:51.881700    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:51.960480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:51.881757    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:52.003478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:51.988797    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:52.199017 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:52.139382    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:52.340493 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:52.316384    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" podUID="6e20c97b-66ce-485b-ba8d-90c2d9e1fa76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:52.341059 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:52.341004    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 30 20:09:52.695896 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:52.695841    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:52.803479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:52.778550    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 20:09:52.868165 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:52.868114    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:52.868535 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:52.868502    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:52.919471 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:52.908158    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf" podUID="a1551c50-4d89-4227-838c-4644c41de49b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 20:09:52.920608 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:52.908615    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:52.944636 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:52.944598    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:09:52.944889 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:52.944862    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf" podUID="48c11b72-fdc6-4d2c-849b-136a5a53259d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:52.959438 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:52.959336    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:52.981591 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:52.959672    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:53.012835 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:52.977149    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:53.157479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:53.156781    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:53.181595 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:53.162041    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 20:09:53.352513 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:53.347797    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" podUID="6e20c97b-66ce-485b-ba8d-90c2d9e1fa76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded"
Oct 30 20:09:53.427516 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:53.425792    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 30 20:09:53.597630 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:53.597585    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf" podUID="a1551c50-4d89-4227-838c-4644c41de49b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:53.806158 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:53.806045    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 20:09:53.874880 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:53.874822    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:53.938008 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:53.937949    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:53.962478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:53.938536    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf" podUID="a1551c50-4d89-4227-838c-4644c41de49b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded"
Oct 30 20:09:53.994142 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:53.994101    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Oct 30 20:09:54.022478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:54.016498    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:54.026908 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:54.018568    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:54.188578 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:54.188528    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:54.302476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:54.209842    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:54.351611 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:54.351568    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n" podUID="6e20c97b-66ce-485b-ba8d-90c2d9e1fa76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/ready\": context deadline exceeded"
Oct 30 20:09:54.382670 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:54.352065    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:09:54.455873 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:54.432058    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:54.835974 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:54.835930    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 20:09:54.885182 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:54.842091    5958 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:09:54.885182 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:54.843949    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 20:09:54.885182 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:54.883067    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n"
Oct 30 20:09:54.909408 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:54.908258    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:09:54.911144 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:54.911122    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:09:54.911374 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:54.911361    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 20:09:54.911539 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:54.911527    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 20:09:54.934069 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:54.929124    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:09:54.934069 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:54.929894    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:09:54.934069 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:54.931663    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:09:54.934069 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:09:54.933120    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:10:06.607730 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:10:06.607692    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:10:09.646133 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:09.646094    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g"]
Oct 30 20:10:09.652110 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:09.652086    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g"
Oct 30 20:10:09.708611 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:09.708575    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g"]
Oct 30 20:10:09.760731 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:09.760699    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-770dfe\""
Oct 30 20:10:09.761111 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:09.761089    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-770dfe-dockercfg-s5b26\""
Oct 30 20:10:09.803473 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:09.785532    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnswf\" (UniqueName: \"kubernetes.io/projected/be48f946-80c5-464d-bd18-408c6d239046-kube-api-access-lnswf\") pod \"payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g\" (UID: \"be48f946-80c5-464d-bd18-408c6d239046\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g"
Oct 30 20:10:09.822474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:09.818933    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be48f946-80c5-464d-bd18-408c6d239046-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g\" (UID: \"be48f946-80c5-464d-bd18-408c6d239046\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g"
Oct 30 20:10:09.845542 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:09.845504    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7"]
Oct 30 20:10:09.852350 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:09.852324    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7"
Oct 30 20:10:09.889954 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:09.889917    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7"]
Oct 30 20:10:09.947522 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:09.929610    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be48f946-80c5-464d-bd18-408c6d239046-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g\" (UID: \"be48f946-80c5-464d-bd18-408c6d239046\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g"
Oct 30 20:10:09.947522 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:09.929703    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnswf\" (UniqueName: \"kubernetes.io/projected/be48f946-80c5-464d-bd18-408c6d239046-kube-api-access-lnswf\") pod \"payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g\" (UID: \"be48f946-80c5-464d-bd18-408c6d239046\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g"
Oct 30 20:10:09.947522 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:09.931341    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be48f946-80c5-464d-bd18-408c6d239046-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g\" (UID: \"be48f946-80c5-464d-bd18-408c6d239046\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g"
Oct 30 20:10:09.992231 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:09.992204    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnswf\" (UniqueName: \"kubernetes.io/projected/be48f946-80c5-464d-bd18-408c6d239046-kube-api-access-lnswf\") pod \"payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g\" (UID: \"be48f946-80c5-464d-bd18-408c6d239046\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g"
Oct 30 20:10:10.031138 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:10.031108    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkwdm\" (UniqueName: \"kubernetes.io/projected/fc5eb38c-c314-4a32-846f-d5b63f2ad009-kube-api-access-vkwdm\") pod \"payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7\" (UID: \"fc5eb38c-c314-4a32-846f-d5b63f2ad009\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7"
Oct 30 20:10:10.031344 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:10.031312    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5eb38c-c314-4a32-846f-d5b63f2ad009-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7\" (UID: \"fc5eb38c-c314-4a32-846f-d5b63f2ad009\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7"
Oct 30 20:10:10.132648 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:10.132605    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkwdm\" (UniqueName: \"kubernetes.io/projected/fc5eb38c-c314-4a32-846f-d5b63f2ad009-kube-api-access-vkwdm\") pod \"payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7\" (UID: \"fc5eb38c-c314-4a32-846f-d5b63f2ad009\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7"
Oct 30 20:10:10.132865 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:10.132845    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5eb38c-c314-4a32-846f-d5b63f2ad009-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7\" (UID: \"fc5eb38c-c314-4a32-846f-d5b63f2ad009\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7"
Oct 30 20:10:10.133896 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:10.133872    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5eb38c-c314-4a32-846f-d5b63f2ad009-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7\" (UID: \"fc5eb38c-c314-4a32-846f-d5b63f2ad009\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7"
Oct 30 20:10:10.194629 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:10.194603    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkwdm\" (UniqueName: \"kubernetes.io/projected/fc5eb38c-c314-4a32-846f-d5b63f2ad009-kube-api-access-vkwdm\") pod \"payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7\" (UID: \"fc5eb38c-c314-4a32-846f-d5b63f2ad009\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7"
Oct 30 20:10:10.199702 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:10.199651    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7"
Oct 30 20:10:10.266944 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:10.266907    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g"
Oct 30 20:10:10.580563 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:10.580539    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7"]
Oct 30 20:10:10.580943 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:10:10.580922    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5eb38c_c314_4a32_846f_d5b63f2ad009.slice/crio-f998643467e14f34c80a0bc2904620f8960d11ab30934e1d96317b6314f3f944 WatchSource:0}: Error finding container f998643467e14f34c80a0bc2904620f8960d11ab30934e1d96317b6314f3f944: Status 404 returned error can't find the container with id f998643467e14f34c80a0bc2904620f8960d11ab30934e1d96317b6314f3f944
Oct 30 20:10:10.693121 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:10.693091    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g"]
Oct 30 20:10:10.694601 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:10:10.694578    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe48f946_80c5_464d_bd18_408c6d239046.slice/crio-81161ec3a767d2e30b3a65c803a4613a6342265e276d140a3c595ffb2387ec26 WatchSource:0}: Error finding container 81161ec3a767d2e30b3a65c803a4613a6342265e276d140a3c595ffb2387ec26: Status 404 returned error can't find the container with id 81161ec3a767d2e30b3a65c803a4613a6342265e276d140a3c595ffb2387ec26
Oct 30 20:10:10.785731 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:10.785699    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525"]
Oct 30 20:10:10.790416 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:10.790396    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525"
Oct 30 20:10:10.860914 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:10.860435    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-27915880-serving-cert\""
Oct 30 20:10:10.866094 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:10.866066    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525"]
Oct 30 20:10:10.945793 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:10.945743    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b8f2b7f-2e80-4fde-ac00-ee68816804be-openshift-service-ca\") pod \"maas-api-e2e-migrate-27915880-6489d6dc7-sb525\" (UID: \"7b8f2b7f-2e80-4fde-ac00-ee68816804be\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525"
Oct 30 20:10:10.945947 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:10.945800    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b8f2b7f-2e80-4fde-ac00-ee68816804be-maas-api-tls\") pod \"maas-api-e2e-migrate-27915880-6489d6dc7-sb525\" (UID: \"7b8f2b7f-2e80-4fde-ac00-ee68816804be\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525"
Oct 30 20:10:10.946079 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:10.946055    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrqjk\" (UniqueName: \"kubernetes.io/projected/7b8f2b7f-2e80-4fde-ac00-ee68816804be-kube-api-access-zrqjk\") pod \"maas-api-e2e-migrate-27915880-6489d6dc7-sb525\" (UID: \"7b8f2b7f-2e80-4fde-ac00-ee68816804be\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525"
Oct 30 20:10:10.946143 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:10.946130    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b8f2b7f-2e80-4fde-ac00-ee68816804be-metrics-certs\") pod \"maas-api-e2e-migrate-27915880-6489d6dc7-sb525\" (UID: \"7b8f2b7f-2e80-4fde-ac00-ee68816804be\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525"
Oct 30 20:10:10.992064 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:10.992020    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g" event={"ID":"be48f946-80c5-464d-bd18-408c6d239046","Type":"ContainerStarted","Data":"a735774fd912e669f85d760567391a6ebb5fdaa378124d5b867b82236885ddec"}
Oct 30 20:10:10.992208 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:10.992082    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g" event={"ID":"be48f946-80c5-464d-bd18-408c6d239046","Type":"ContainerStarted","Data":"81161ec3a767d2e30b3a65c803a4613a6342265e276d140a3c595ffb2387ec26"}
Oct 30 20:10:10.992208 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:10.992138    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g"
Oct 30 20:10:10.994015 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:10.993985    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7" event={"ID":"fc5eb38c-c314-4a32-846f-d5b63f2ad009","Type":"ContainerStarted","Data":"42583b5c359fd37e713f7b451ca4d2773358a7451fdf6e8133c8b14e55f02cc4"}
Oct 30 20:10:10.994131 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:10.994028    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7" event={"ID":"fc5eb38c-c314-4a32-846f-d5b63f2ad009","Type":"ContainerStarted","Data":"f998643467e14f34c80a0bc2904620f8960d11ab30934e1d96317b6314f3f944"}
Oct 30 20:10:10.994131 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:10.994047    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7"
Oct 30 20:10:11.048248 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:11.048217    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b8f2b7f-2e80-4fde-ac00-ee68816804be-openshift-service-ca\") pod \"maas-api-e2e-migrate-27915880-6489d6dc7-sb525\" (UID: \"7b8f2b7f-2e80-4fde-ac00-ee68816804be\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525"
Oct 30 20:10:11.048500 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:11.048267    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b8f2b7f-2e80-4fde-ac00-ee68816804be-maas-api-tls\") pod \"maas-api-e2e-migrate-27915880-6489d6dc7-sb525\" (UID: \"7b8f2b7f-2e80-4fde-ac00-ee68816804be\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525"
Oct 30 20:10:11.048588 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:11.048536    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrqjk\" (UniqueName: \"kubernetes.io/projected/7b8f2b7f-2e80-4fde-ac00-ee68816804be-kube-api-access-zrqjk\") pod \"maas-api-e2e-migrate-27915880-6489d6dc7-sb525\" (UID: \"7b8f2b7f-2e80-4fde-ac00-ee68816804be\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525"
Oct 30 20:10:11.048649 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:11.048618    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b8f2b7f-2e80-4fde-ac00-ee68816804be-metrics-certs\") pod \"maas-api-e2e-migrate-27915880-6489d6dc7-sb525\" (UID: \"7b8f2b7f-2e80-4fde-ac00-ee68816804be\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525"
Oct 30 20:10:11.049260 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:10:11.049233    5958 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-serving-cert: secret "maas-api-e2e-migrate-27915880-serving-cert" not found
Oct 30 20:10:11.049405 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:10:11.049319    5958 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b8f2b7f-2e80-4fde-ac00-ee68816804be-maas-api-tls podName:7b8f2b7f-2e80-4fde-ac00-ee68816804be nodeName:}" failed. No retries permitted until 2026-10-30 20:10:11.54929778 +0000 UTC m=+1650.281513167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7b8f2b7f-2e80-4fde-ac00-ee68816804be-maas-api-tls") pod "maas-api-e2e-migrate-27915880-6489d6dc7-sb525" (UID: "7b8f2b7f-2e80-4fde-ac00-ee68816804be") : secret "maas-api-e2e-migrate-27915880-serving-cert" not found
Oct 30 20:10:11.050147 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:11.050126    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b8f2b7f-2e80-4fde-ac00-ee68816804be-openshift-service-ca\") pod \"maas-api-e2e-migrate-27915880-6489d6dc7-sb525\" (UID: \"7b8f2b7f-2e80-4fde-ac00-ee68816804be\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525"
Oct 30 20:10:11.051817 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:11.051794    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b8f2b7f-2e80-4fde-ac00-ee68816804be-metrics-certs\") pod \"maas-api-e2e-migrate-27915880-6489d6dc7-sb525\" (UID: \"7b8f2b7f-2e80-4fde-ac00-ee68816804be\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525"
Oct 30 20:10:11.118901 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:11.118805    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g" podStartSLOduration=2.118791932 podStartE2EDuration="2.118791932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:11.114400016 +0000 UTC m=+1649.846615403" watchObservedRunningTime="2026-10-30 20:10:11.118791932 +0000 UTC m=+1649.851007320"
Oct 30 20:10:11.124557 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:11.124528    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrqjk\" (UniqueName: \"kubernetes.io/projected/7b8f2b7f-2e80-4fde-ac00-ee68816804be-kube-api-access-zrqjk\") pod \"maas-api-e2e-migrate-27915880-6489d6dc7-sb525\" (UID: \"7b8f2b7f-2e80-4fde-ac00-ee68816804be\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525"
Oct 30 20:10:11.234137 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:11.234082    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7" podStartSLOduration=2.234067408 podStartE2EDuration="2.234067408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:11.232489926 +0000 UTC m=+1649.964705321" watchObservedRunningTime="2026-10-30 20:10:11.234067408 +0000 UTC m=+1649.966282799"
Oct 30 20:10:11.557011 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:11.556926    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b8f2b7f-2e80-4fde-ac00-ee68816804be-maas-api-tls\") pod \"maas-api-e2e-migrate-27915880-6489d6dc7-sb525\" (UID: \"7b8f2b7f-2e80-4fde-ac00-ee68816804be\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525"
Oct 30 20:10:11.559592 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:11.559564    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b8f2b7f-2e80-4fde-ac00-ee68816804be-maas-api-tls\") pod \"maas-api-e2e-migrate-27915880-6489d6dc7-sb525\" (UID: \"7b8f2b7f-2e80-4fde-ac00-ee68816804be\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525"
Oct 30 20:10:11.705405 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:11.705368    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525"
Oct 30 20:10:12.101085 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:12.101048    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525"]
Oct 30 20:10:12.101247 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:10:12.101225    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8f2b7f_2e80_4fde_ac00_ee68816804be.slice/crio-84c10798cac4578138bf84c458b9f6e3726e171aaa0d66fe2f431e3faff3671b WatchSource:0}: Error finding container 84c10798cac4578138bf84c458b9f6e3726e171aaa0d66fe2f431e3faff3671b: Status 404 returned error can't find the container with id 84c10798cac4578138bf84c458b9f6e3726e171aaa0d66fe2f431e3faff3671b
Oct 30 20:10:13.012665 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:13.012631    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525" event={"ID":"7b8f2b7f-2e80-4fde-ac00-ee68816804be","Type":"ContainerStarted","Data":"84c10798cac4578138bf84c458b9f6e3726e171aaa0d66fe2f431e3faff3671b"}
Oct 30 20:10:14.022884 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:14.022842    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525" event={"ID":"7b8f2b7f-2e80-4fde-ac00-ee68816804be","Type":"ContainerStarted","Data":"1a1b093d8f5968e785fabd63cd39a88bb85893d7317276073bf0a550c45b4f2e"}
Oct 30 20:10:14.023265 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:14.022926    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525"
Oct 30 20:10:14.154562 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:14.154510    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525" podStartSLOduration=2.944115447 podStartE2EDuration="4.154495457s" totalImagesPullingTime="1.21038001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:14.152507882 +0000 UTC m=+1652.884723272" watchObservedRunningTime="2026-10-30 20:10:14.154495457 +0000 UTC m=+1652.886710844"
Oct 30 20:10:20.039040 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:20.039006    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525"
Oct 30 20:10:22.005283 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:22.005252    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g"
Oct 30 20:10:22.005699 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:22.005330    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7"
Oct 30 20:10:23.954654 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:23.954617    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"]
Oct 30 20:10:23.959524 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:23.959506    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.066921 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.066879    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.067910 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.067873    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.068052 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.067918    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b3d965b-503e-44f3-b95d-b7c599310b81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.068052 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.067989    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dtm4\" (UniqueName: \"kubernetes.io/projected/3b3d965b-503e-44f3-b95d-b7c599310b81-kube-api-access-5dtm4\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.068052 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.068020    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.068245 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.068052    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.068245 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.068189    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.068245 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.068220    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.068410 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.068273    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.073052 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.073027    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"]
Oct 30 20:10:24.170444 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.170406    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.170640 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.170510    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.170640 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.170541    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b3d965b-503e-44f3-b95d-b7c599310b81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.170640 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.170611    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dtm4\" (UniqueName: \"kubernetes.io/projected/3b3d965b-503e-44f3-b95d-b7c599310b81-kube-api-access-5dtm4\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.170857 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.170643    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.170857 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.170675    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.170857 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.170815    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.170857 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.170849    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.171064 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.170903    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.171064 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.170955    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.171493 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.171440    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.171721 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.171653    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.171856 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.171829    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.172377 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.172351    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b3d965b-503e-44f3-b95d-b7c599310b81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.173413 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.173390    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.174188 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.174166    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.234126 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.234057    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.237775 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.237743    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dtm4\" (UniqueName: \"kubernetes.io/projected/3b3d965b-503e-44f3-b95d-b7c599310b81-kube-api-access-5dtm4\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-w8zf4\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.273809 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.273778    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:24.770480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.762873    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:10:24.770480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.762962    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:10:24.770480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.763004    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:10:24.770480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:24.765075    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"]
Oct 30 20:10:25.134479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:25.133764    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4" event={"ID":"3b3d965b-503e-44f3-b95d-b7c599310b81","Type":"ContainerStarted","Data":"b2e55881e5f45d47e7da5fb443c1edc0aa0abc8a738898cb087e57364b63506e"}
Oct 30 20:10:25.134479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:25.133839    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4" event={"ID":"3b3d965b-503e-44f3-b95d-b7c599310b81","Type":"ContainerStarted","Data":"8247d47a9a06d74fcc6be1204e1a3258ef6d6f3e913ae413e2c7429457444e7f"}
Oct 30 20:10:25.280478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:25.279986    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4" podStartSLOduration=2.279968112 podStartE2EDuration="2.279968112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:25.272073457 +0000 UTC m=+1664.004288848" watchObservedRunningTime="2026-10-30 20:10:25.279968112 +0000 UTC m=+1664.012183503"
Oct 30 20:10:25.287042 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:25.283348    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:25.287042 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:25.283807    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 20:10:25.287042 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:25.283880    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4" podUID="3b3d965b-503e-44f3-b95d-b7c599310b81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 20:10:26.287476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:26.286997    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 20:10:26.287476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:26.287058    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4" podUID="3b3d965b-503e-44f3-b95d-b7c599310b81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 20:10:27.286737 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:27.286700    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 20:10:27.287032 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:27.286998    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4" podUID="3b3d965b-503e-44f3-b95d-b7c599310b81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 20:10:27.666780 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:27.666740    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn"]
Oct 30 20:10:27.675886 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:27.675859    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn"
Oct 30 20:10:27.713650 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:27.713609    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn"]
Oct 30 20:10:27.767605 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:27.759926    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d7ca5342-serving-cert\""
Oct 30 20:10:27.772938 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:27.772899    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90ae88c8-fb47-43da-8804-c10a62cb2179-metrics-certs\") pod \"maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn\" (UID: \"90ae88c8-fb47-43da-8804-c10a62cb2179\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn"
Oct 30 20:10:27.773186 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:27.773151    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ae88c8-fb47-43da-8804-c10a62cb2179-maas-api-tls\") pod \"maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn\" (UID: \"90ae88c8-fb47-43da-8804-c10a62cb2179\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn"
Oct 30 20:10:27.773259 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:27.773200    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ae88c8-fb47-43da-8804-c10a62cb2179-openshift-service-ca\") pod \"maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn\" (UID: \"90ae88c8-fb47-43da-8804-c10a62cb2179\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn"
Oct 30 20:10:27.773543 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:27.773510    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mxfv\" (UniqueName: \"kubernetes.io/projected/90ae88c8-fb47-43da-8804-c10a62cb2179-kube-api-access-9mxfv\") pod \"maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn\" (UID: \"90ae88c8-fb47-43da-8804-c10a62cb2179\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn"
Oct 30 20:10:27.874730 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:27.874691    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ae88c8-fb47-43da-8804-c10a62cb2179-maas-api-tls\") pod \"maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn\" (UID: \"90ae88c8-fb47-43da-8804-c10a62cb2179\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn"
Oct 30 20:10:27.874912 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:27.874740    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ae88c8-fb47-43da-8804-c10a62cb2179-openshift-service-ca\") pod \"maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn\" (UID: \"90ae88c8-fb47-43da-8804-c10a62cb2179\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn"
Oct 30 20:10:27.874912 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:10:27.874830    5958 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-serving-cert: secret "maas-api-e2e-ait-d7ca5342-serving-cert" not found
Oct 30 20:10:27.874912 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:27.874867    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mxfv\" (UniqueName: \"kubernetes.io/projected/90ae88c8-fb47-43da-8804-c10a62cb2179-kube-api-access-9mxfv\") pod \"maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn\" (UID: \"90ae88c8-fb47-43da-8804-c10a62cb2179\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn"
Oct 30 20:10:27.874912 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:10:27.874898    5958 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90ae88c8-fb47-43da-8804-c10a62cb2179-maas-api-tls podName:90ae88c8-fb47-43da-8804-c10a62cb2179 nodeName:}" failed. No retries permitted until 2026-10-30 20:10:28.374882704 +0000 UTC m=+1667.107098073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/90ae88c8-fb47-43da-8804-c10a62cb2179-maas-api-tls") pod "maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn" (UID: "90ae88c8-fb47-43da-8804-c10a62cb2179") : secret "maas-api-e2e-ait-d7ca5342-serving-cert" not found
Oct 30 20:10:27.875124 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:27.875005    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90ae88c8-fb47-43da-8804-c10a62cb2179-metrics-certs\") pod \"maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn\" (UID: \"90ae88c8-fb47-43da-8804-c10a62cb2179\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn"
Oct 30 20:10:27.875436 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:27.875415    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ae88c8-fb47-43da-8804-c10a62cb2179-openshift-service-ca\") pod \"maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn\" (UID: \"90ae88c8-fb47-43da-8804-c10a62cb2179\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn"
Oct 30 20:10:27.877614 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:27.877594    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90ae88c8-fb47-43da-8804-c10a62cb2179-metrics-certs\") pod \"maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn\" (UID: \"90ae88c8-fb47-43da-8804-c10a62cb2179\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn"
Oct 30 20:10:27.958802 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:27.958680    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mxfv\" (UniqueName: \"kubernetes.io/projected/90ae88c8-fb47-43da-8804-c10a62cb2179-kube-api-access-9mxfv\") pod \"maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn\" (UID: \"90ae88c8-fb47-43da-8804-c10a62cb2179\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn"
Oct 30 20:10:28.275059 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:28.274962    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 20:10:28.275059 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:28.275020    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4" podUID="3b3d965b-503e-44f3-b95d-b7c599310b81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 20:10:28.383953 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:28.383914    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ae88c8-fb47-43da-8804-c10a62cb2179-maas-api-tls\") pod \"maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn\" (UID: \"90ae88c8-fb47-43da-8804-c10a62cb2179\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn"
Oct 30 20:10:28.386516 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:28.386487    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ae88c8-fb47-43da-8804-c10a62cb2179-maas-api-tls\") pod \"maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn\" (UID: \"90ae88c8-fb47-43da-8804-c10a62cb2179\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn"
Oct 30 20:10:28.594700 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:28.594665    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn"
Oct 30 20:10:29.012498 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:29.012439    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn"]
Oct 30 20:10:29.012831 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:10:29.012651    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ae88c8_fb47_43da_8804_c10a62cb2179.slice/crio-44c74a02584f947540e46c6e54f7a131561d0eec8bfb3cc4e8caf9f8e5392677 WatchSource:0}: Error finding container 44c74a02584f947540e46c6e54f7a131561d0eec8bfb3cc4e8caf9f8e5392677: Status 404 returned error can't find the container with id 44c74a02584f947540e46c6e54f7a131561d0eec8bfb3cc4e8caf9f8e5392677
Oct 30 20:10:29.246910 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:29.246863    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn" event={"ID":"90ae88c8-fb47-43da-8804-c10a62cb2179","Type":"ContainerStarted","Data":"44c74a02584f947540e46c6e54f7a131561d0eec8bfb3cc4e8caf9f8e5392677"}
Oct 30 20:10:29.274218 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:29.274132    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 20:10:29.274218 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:29.274190    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4" podUID="3b3d965b-503e-44f3-b95d-b7c599310b81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 20:10:30.267085 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:30.267044    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn" event={"ID":"90ae88c8-fb47-43da-8804-c10a62cb2179","Type":"ContainerStarted","Data":"c25fff084a384832a3c9796073966dd8b0a83b2931e2117dd89d6b76ff9e1e66"}
Oct 30 20:10:30.267513 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:30.267142    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn"
Oct 30 20:10:30.274062 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:30.274035    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 20:10:30.274195 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:30.274085    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4" podUID="3b3d965b-503e-44f3-b95d-b7c599310b81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 20:10:30.387166 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:30.387109    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn" podStartSLOduration=2.604438914 podStartE2EDuration="3.387093345s" totalImagesPullingTime="782.654431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:30.385034622 +0000 UTC m=+1669.117250013" watchObservedRunningTime="2026-10-30 20:10:30.387093345 +0000 UTC m=+1669.119308733"
Oct 30 20:10:31.274056 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:31.274022    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 20:10:31.274688 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:31.274085    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4" podUID="3b3d965b-503e-44f3-b95d-b7c599310b81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 20:10:32.274609 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:32.274570    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 20:10:32.274997 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:32.274639    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4" podUID="3b3d965b-503e-44f3-b95d-b7c599310b81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 20:10:33.554637 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:33.554597    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:33.555567 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:33.555521    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:33.556626 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:33.556601    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:10:36.349205 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:36.335627    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn"
Oct 30 20:10:36.607401 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:10:36.607308    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:10:41.731374 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:41.731335    5958 scope.go:122] "RemoveContainer" containerID="395648b038f0b43aeee8938954d7a04fd7241a9e1e9a87d4e89f5ab372b85dad"
Oct 30 20:10:41.742025 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:10:41.742001    5958 scope.go:122] "RemoveContainer" containerID="d0b8a5e1570b2bf9169cd752a29b31c41a453f37d862360f794d7201b9868132"
Oct 30 20:11:00.578925 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:00.578888    5958 generic.go:356] "Generic (PLEG): container finished" podID="ede036c9-86b3-4d4b-95b0-88570503f1c2" containerID="f5f7b43196475efd0db9db8770deabe65b8a5276355d90df18320252ede3dbb2" exitCode=0
Oct 30 20:11:00.579406 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:00.578960    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c" event={"ID":"ede036c9-86b3-4d4b-95b0-88570503f1c2","Type":"ContainerDied","Data":"f5f7b43196475efd0db9db8770deabe65b8a5276355d90df18320252ede3dbb2"}
Oct 30 20:11:00.579616 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:00.579596    5958 scope.go:122] "RemoveContainer" containerID="f5f7b43196475efd0db9db8770deabe65b8a5276355d90df18320252ede3dbb2"
Oct 30 20:11:00.581397 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:00.581368    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:11:00.581490 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:00.581438    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:11:00.581533 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:00.581494    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:11:01.589402 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:01.589369    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c" event={"ID":"ede036c9-86b3-4d4b-95b0-88570503f1c2","Type":"ContainerStarted","Data":"ed6100f1fb6f45d55fe247130b951bd59c0285dbc9c9b308e742ee37d90997c6"}
Oct 30 20:11:01.591799 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:01.591771    5958 generic.go:356] "Generic (PLEG): container finished" podID="33bf491a-644a-4a04-b8d5-5ca6842ff3e6" containerID="5041afe3a7604006b478cc2e63cf910ce0291f8f47d04fc298fa48ad9f16383b" exitCode=0
Oct 30 20:11:01.591924 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:01.591841    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6" event={"ID":"33bf491a-644a-4a04-b8d5-5ca6842ff3e6","Type":"ContainerDied","Data":"5041afe3a7604006b478cc2e63cf910ce0291f8f47d04fc298fa48ad9f16383b"}
Oct 30 20:11:01.592239 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:01.592225    5958 scope.go:122] "RemoveContainer" containerID="5041afe3a7604006b478cc2e63cf910ce0291f8f47d04fc298fa48ad9f16383b"
Oct 30 20:11:01.593774 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:01.593742    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:11:01.593885 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:01.593828    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:11:01.593885 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:01.593877    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:11:02.261212 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:02.260951    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:11:02.262112 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:02.262077    5958 patch_prober.go:29] interesting pod/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:11:02.262235 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:02.262144    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c" podUID="ede036c9-86b3-4d4b-95b0-88570503f1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 20:11:02.603322 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:02.603277    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6" event={"ID":"33bf491a-644a-4a04-b8d5-5ca6842ff3e6","Type":"ContainerStarted","Data":"c863b30ecd2a6d8b69f96b0024dee9c2bd8173efe8d428142cf68d582ca329d6"}
Oct 30 20:11:02.730089 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:02.730051    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"]
Oct 30 20:11:02.736548 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:02.734817    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"
Oct 30 20:11:02.787737 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:02.787700    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-770dfe-serving-cert\""
Oct 30 20:11:02.791275 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:02.791192    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b98921-b374-47ca-93d1-6ccd66cae325-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr\" (UID: \"b8b98921-b374-47ca-93d1-6ccd66cae325\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"
Oct 30 20:11:02.791860 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:02.791824    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b98921-b374-47ca-93d1-6ccd66cae325-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr\" (UID: \"b8b98921-b374-47ca-93d1-6ccd66cae325\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"
Oct 30 20:11:02.792607 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:02.792571    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8b98921-b374-47ca-93d1-6ccd66cae325-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr\" (UID: \"b8b98921-b374-47ca-93d1-6ccd66cae325\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"
Oct 30 20:11:02.792867 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:02.792848    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjkn6\" (UniqueName: \"kubernetes.io/projected/b8b98921-b374-47ca-93d1-6ccd66cae325-kube-api-access-hjkn6\") pod \"maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr\" (UID: \"b8b98921-b374-47ca-93d1-6ccd66cae325\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"
Oct 30 20:11:02.804533 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:02.804340    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"]
Oct 30 20:11:02.898572 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:02.895680    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8b98921-b374-47ca-93d1-6ccd66cae325-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr\" (UID: \"b8b98921-b374-47ca-93d1-6ccd66cae325\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"
Oct 30 20:11:02.898572 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:02.895785    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjkn6\" (UniqueName: \"kubernetes.io/projected/b8b98921-b374-47ca-93d1-6ccd66cae325-kube-api-access-hjkn6\") pod \"maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr\" (UID: \"b8b98921-b374-47ca-93d1-6ccd66cae325\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"
Oct 30 20:11:02.898572 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:02.895978    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b98921-b374-47ca-93d1-6ccd66cae325-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr\" (UID: \"b8b98921-b374-47ca-93d1-6ccd66cae325\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"
Oct 30 20:11:02.898572 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:02.896134    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b98921-b374-47ca-93d1-6ccd66cae325-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr\" (UID: \"b8b98921-b374-47ca-93d1-6ccd66cae325\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"
Oct 30 20:11:02.898572 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:11:02.897581    5958 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-serving-cert: secret "maas-api-e2e-shared-a-w2-770dfe-serving-cert" not found
Oct 30 20:11:02.898572 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:11:02.897655    5958 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8b98921-b374-47ca-93d1-6ccd66cae325-maas-api-tls podName:b8b98921-b374-47ca-93d1-6ccd66cae325 nodeName:}" failed. No retries permitted until 2026-10-30 20:11:03.397633808 +0000 UTC m=+1702.129849193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8b98921-b374-47ca-93d1-6ccd66cae325-maas-api-tls") pod "maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr" (UID: "b8b98921-b374-47ca-93d1-6ccd66cae325") : secret "maas-api-e2e-shared-a-w2-770dfe-serving-cert" not found
Oct 30 20:11:02.898572 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:02.898135    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b98921-b374-47ca-93d1-6ccd66cae325-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr\" (UID: \"b8b98921-b374-47ca-93d1-6ccd66cae325\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"
Oct 30 20:11:02.900489 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:02.900216    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8b98921-b374-47ca-93d1-6ccd66cae325-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr\" (UID: \"b8b98921-b374-47ca-93d1-6ccd66cae325\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"
Oct 30 20:11:03.011162 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:03.011112    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjkn6\" (UniqueName: \"kubernetes.io/projected/b8b98921-b374-47ca-93d1-6ccd66cae325-kube-api-access-hjkn6\") pod \"maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr\" (UID: \"b8b98921-b374-47ca-93d1-6ccd66cae325\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"
Oct 30 20:11:03.198872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:03.198607    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:11:03.198872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:03.198814    5958 patch_prober.go:29] interesting pod/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 20:11:03.199103 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:03.198863    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6" podUID="33bf491a-644a-4a04-b8d5-5ca6842ff3e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 20:11:03.257476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:03.253860    5958 patch_prober.go:29] interesting pod/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:11:03.257476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:03.253922    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c" podUID="ede036c9-86b3-4d4b-95b0-88570503f1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 20:11:03.416332 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:03.416297    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b98921-b374-47ca-93d1-6ccd66cae325-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr\" (UID: \"b8b98921-b374-47ca-93d1-6ccd66cae325\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"
Oct 30 20:11:03.416549 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:11:03.416476    5958 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-serving-cert: secret "maas-api-e2e-shared-a-w2-770dfe-serving-cert" not found
Oct 30 20:11:03.416627 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:11:03.416548    5958 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8b98921-b374-47ca-93d1-6ccd66cae325-maas-api-tls podName:b8b98921-b374-47ca-93d1-6ccd66cae325 nodeName:}" failed. No retries permitted until 2026-10-30 20:11:04.416528586 +0000 UTC m=+1703.148743968 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8b98921-b374-47ca-93d1-6ccd66cae325-maas-api-tls") pod "maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr" (UID: "b8b98921-b374-47ca-93d1-6ccd66cae325") : secret "maas-api-e2e-shared-a-w2-770dfe-serving-cert" not found
Oct 30 20:11:04.196430 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:04.196399    5958 patch_prober.go:29] interesting pod/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 20:11:04.196901 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:04.196874    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6" podUID="33bf491a-644a-4a04-b8d5-5ca6842ff3e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 20:11:04.255476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:04.255404    5958 patch_prober.go:29] interesting pod/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:11:04.255684 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:04.255519    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c" podUID="ede036c9-86b3-4d4b-95b0-88570503f1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 20:11:04.395367 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:04.395313    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:11:04.433842 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:04.433813    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b98921-b374-47ca-93d1-6ccd66cae325-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr\" (UID: \"b8b98921-b374-47ca-93d1-6ccd66cae325\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"
Oct 30 20:11:04.458530 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:04.454057    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b98921-b374-47ca-93d1-6ccd66cae325-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr\" (UID: \"b8b98921-b374-47ca-93d1-6ccd66cae325\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"
Oct 30 20:11:04.598924 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:04.598891    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"
Oct 30 20:11:05.099885 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:05.099851    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"]
Oct 30 20:11:05.196628 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:05.196587    5958 patch_prober.go:29] interesting pod/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 20:11:05.197647 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:05.197191    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6" podUID="33bf491a-644a-4a04-b8d5-5ca6842ff3e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 20:11:05.260478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:05.259570    5958 patch_prober.go:29] interesting pod/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:11:05.260478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:05.259637    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c" podUID="ede036c9-86b3-4d4b-95b0-88570503f1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 20:11:05.403278 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:05.403199    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:11:05.630190 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:05.630159    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr" event={"ID":"b8b98921-b374-47ca-93d1-6ccd66cae325","Type":"ContainerStarted","Data":"5d6135e453f15d55d70dfc695e4c67ef0b559d1cb1ed612b23cd8fca6fc42c37"}
Oct 30 20:11:06.196213 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:06.196173    5958 patch_prober.go:29] interesting pod/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 20:11:06.196356 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:06.196247    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6" podUID="33bf491a-644a-4a04-b8d5-5ca6842ff3e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 20:11:06.275529 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:06.265809    5958 patch_prober.go:29] interesting pod/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 20:11:06.275529 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:06.274672    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c" podUID="ede036c9-86b3-4d4b-95b0-88570503f1c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 20:11:06.667316 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:06.667272    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr" event={"ID":"b8b98921-b374-47ca-93d1-6ccd66cae325","Type":"ContainerStarted","Data":"054be163d5c50ff24e5873d8df6d6c523e4fb3f85c95672473e4a64df673dfb3"}
Oct 30 20:11:06.683161 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:06.683128    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"
Oct 30 20:11:06.696176 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:11:06.696139    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:11:06.799979 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:06.799921    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr" podStartSLOduration=3.891294855 podStartE2EDuration="4.799904771s" totalImagesPullingTime="908.609916ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:06.796959093 +0000 UTC m=+1705.529174496" watchObservedRunningTime="2026-10-30 20:11:06.799904771 +0000 UTC m=+1705.532120161"
Oct 30 20:11:07.196497 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:07.196440    5958 patch_prober.go:29] interesting pod/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 20:11:07.196680 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:07.196517    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6" podUID="33bf491a-644a-4a04-b8d5-5ca6842ff3e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 20:11:07.280009 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:07.279973    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:11:07.281873 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:07.281851    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:11:08.200952 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:08.200918    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:11:08.202118 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:08.202094    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:11:12.688689 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:12.688654    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"
Oct 30 20:11:25.053576 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:25.053488    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb1e9517-jmwhb"]
Oct 30 20:11:25.102908 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:25.102873    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb1e9517-jmwhb"]
Oct 30 20:11:25.431514 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:25.431409    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aac6c75-c3d7-4df7-8b56-f08a89dffe09" path="/var/lib/kubelet/pods/9aac6c75-c3d7-4df7-8b56-f08a89dffe09/volumes"
Oct 30 20:11:30.745215 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:30.745178    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9"]
Oct 30 20:11:30.850159 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:30.850125    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn"]
Oct 30 20:11:35.746030 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:35.745967    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9" podUID="ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4" containerName="payload-processing" containerID="cri-o://b2a191afe3a6b2c43f49fad6a86dfd13449610e684e2873038bb4f106f462c06" gracePeriod=25
Oct 30 20:11:35.851296 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:35.851212    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn" podUID="6530268f-b6ee-4315-ab68-affa9a4057f9" containerName="payload-pre-processing" containerID="cri-o://68bb14775d4f275a5dc211dbc3c4a1da55b69ce02502044281ae7fee4de11236" gracePeriod=25
Oct 30 20:11:35.921389 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:35.921353    5958 generic.go:356] "Generic (PLEG): container finished" podID="ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4" containerID="b2a191afe3a6b2c43f49fad6a86dfd13449610e684e2873038bb4f106f462c06" exitCode=0
Oct 30 20:11:35.921602 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:35.921424    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9" event={"ID":"ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4","Type":"ContainerDied","Data":"b2a191afe3a6b2c43f49fad6a86dfd13449610e684e2873038bb4f106f462c06"}
Oct 30 20:11:36.056004 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.055974    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9"
Oct 30 20:11:36.085152 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.085119    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv"]
Oct 30 20:11:36.131078 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.131043    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4-kube-api-access-2fxpn\" (UniqueName: \"kubernetes.io/projected/ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4-kube-api-access-2fxpn\") pod \"ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4\" (UID: \"ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4\") "
Oct 30 20:11:36.131292 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.131265    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4-plugins-config-volume\") pod \"ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4\" (UID: \"ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4\") "
Oct 30 20:11:36.131721 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.131689    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4-plugins-config-volume" pod "ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4" (UID: "ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:36.132419 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.132394    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:36.134176 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.134130    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4-kube-api-access-2fxpn" pod "ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4" (UID: "ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4"). InnerVolumeSpecName "kube-api-access-2fxpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:36.164268 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.164241    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn"
Oct 30 20:11:36.206010 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.205974    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc"]
Oct 30 20:11:36.235282 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.235242    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fxpn\" (UniqueName: \"kubernetes.io/projected/ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4-kube-api-access-2fxpn\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:36.339884 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.339845    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6530268f-b6ee-4315-ab68-affa9a4057f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6530268f-b6ee-4315-ab68-affa9a4057f9-plugins-config-volume\") pod \"6530268f-b6ee-4315-ab68-affa9a4057f9\" (UID: \"6530268f-b6ee-4315-ab68-affa9a4057f9\") "
Oct 30 20:11:36.340183 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.340156    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6530268f-b6ee-4315-ab68-affa9a4057f9-kube-api-access-cvs2m\" (UniqueName: \"kubernetes.io/projected/6530268f-b6ee-4315-ab68-affa9a4057f9-kube-api-access-cvs2m\") pod \"6530268f-b6ee-4315-ab68-affa9a4057f9\" (UID: \"6530268f-b6ee-4315-ab68-affa9a4057f9\") "
Oct 30 20:11:36.340946 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.340918    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6530268f-b6ee-4315-ab68-affa9a4057f9-plugins-config-volume" pod "6530268f-b6ee-4315-ab68-affa9a4057f9" (UID: "6530268f-b6ee-4315-ab68-affa9a4057f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:36.341346 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.341321    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6530268f-b6ee-4315-ab68-affa9a4057f9-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:36.347802 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.347772    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6530268f-b6ee-4315-ab68-affa9a4057f9-kube-api-access-cvs2m" pod "6530268f-b6ee-4315-ab68-affa9a4057f9" (UID: "6530268f-b6ee-4315-ab68-affa9a4057f9"). InnerVolumeSpecName "kube-api-access-cvs2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:36.443125 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.443088    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvs2m\" (UniqueName: \"kubernetes.io/projected/6530268f-b6ee-4315-ab68-affa9a4057f9-kube-api-access-cvs2m\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:36.615422 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:11:36.615338    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:11:36.931180 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.931091    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9"
Oct 30 20:11:36.931180 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.931109    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9" event={"ID":"ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4","Type":"ContainerDied","Data":"a7f0ddfc832ace5106e258b370552c99041df14aa5df463b6cdf4c4624dda9d8"}
Oct 30 20:11:36.931769 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.931194    5958 scope.go:122] "RemoveContainer" containerID="b2a191afe3a6b2c43f49fad6a86dfd13449610e684e2873038bb4f106f462c06"
Oct 30 20:11:36.932813 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.932791    5958 generic.go:356] "Generic (PLEG): container finished" podID="6530268f-b6ee-4315-ab68-affa9a4057f9" containerID="68bb14775d4f275a5dc211dbc3c4a1da55b69ce02502044281ae7fee4de11236" exitCode=0
Oct 30 20:11:36.932896 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.932837    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn"
Oct 30 20:11:36.932896 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.932869    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn" event={"ID":"6530268f-b6ee-4315-ab68-affa9a4057f9","Type":"ContainerDied","Data":"68bb14775d4f275a5dc211dbc3c4a1da55b69ce02502044281ae7fee4de11236"}
Oct 30 20:11:36.932991 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.932933    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn" event={"ID":"6530268f-b6ee-4315-ab68-affa9a4057f9","Type":"ContainerDied","Data":"20083a301c6852bf5a34302351a01607c8410559f28dc4b59dfe23d0c87fa799"}
Oct 30 20:11:36.948888 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.948860    5958 scope.go:122] "RemoveContainer" containerID="68bb14775d4f275a5dc211dbc3c4a1da55b69ce02502044281ae7fee4de11236"
Oct 30 20:11:36.960201 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.960178    5958 scope.go:122] "RemoveContainer" containerID="68bb14775d4f275a5dc211dbc3c4a1da55b69ce02502044281ae7fee4de11236"
Oct 30 20:11:36.960581 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:11:36.960554    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68bb14775d4f275a5dc211dbc3c4a1da55b69ce02502044281ae7fee4de11236\": container with ID starting with 68bb14775d4f275a5dc211dbc3c4a1da55b69ce02502044281ae7fee4de11236 not found: ID does not exist" containerID="68bb14775d4f275a5dc211dbc3c4a1da55b69ce02502044281ae7fee4de11236"
Oct 30 20:11:36.960644 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:36.960592    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68bb14775d4f275a5dc211dbc3c4a1da55b69ce02502044281ae7fee4de11236"} err="failed to get container status \"68bb14775d4f275a5dc211dbc3c4a1da55b69ce02502044281ae7fee4de11236\": rpc error: code = NotFound desc = could not find container \"68bb14775d4f275a5dc211dbc3c4a1da55b69ce02502044281ae7fee4de11236\": container with ID starting with 68bb14775d4f275a5dc211dbc3c4a1da55b69ce02502044281ae7fee4de11236 not found: ID does not exist"
Oct 30 20:11:37.098815 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:37.098767    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn"]
Oct 30 20:11:37.103500 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:37.103472    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-27915880-6755b647cf-tkvfn"]
Oct 30 20:11:37.213875 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:37.213793    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9"]
Oct 30 20:11:37.218087 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:37.218059    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-27915880-64c5745cb8-bmsj9"]
Oct 30 20:11:37.401031 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:37.400993    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6530268f-b6ee-4315-ab68-affa9a4057f9" path="/var/lib/kubelet/pods/6530268f-b6ee-4315-ab68-affa9a4057f9/volumes"
Oct 30 20:11:37.401582 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:37.401565    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4" path="/var/lib/kubelet/pods/ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4/volumes"
Oct 30 20:11:41.085868 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:41.085813    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv" podUID="026dbd11-0bd0-445d-8e44-388ceaf9418a" containerName="payload-processing" containerID="cri-o://5c382a5097345aad32763d58eaa5fdded497870bbd9942ef8f8ebf947ef3ffa2" gracePeriod=25
Oct 30 20:11:41.207021 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:41.206986    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc" podUID="fc1532ea-e75f-4f6e-9791-8e95bcf51cf8" containerName="payload-pre-processing" containerID="cri-o://2c5caeafd700ae0de212bbc5e4bf1fae2b125a4175e26e3621378f69525e49f8" gracePeriod=25
Oct 30 20:11:41.383078 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:41.382987    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv"
Oct 30 20:11:41.438054 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:41.438023    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/026dbd11-0bd0-445d-8e44-388ceaf9418a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/026dbd11-0bd0-445d-8e44-388ceaf9418a-plugins-config-volume\") pod \"026dbd11-0bd0-445d-8e44-388ceaf9418a\" (UID: \"026dbd11-0bd0-445d-8e44-388ceaf9418a\") "
Oct 30 20:11:41.438215 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:41.438063    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/026dbd11-0bd0-445d-8e44-388ceaf9418a-kube-api-access-br52p\" (UniqueName: \"kubernetes.io/projected/026dbd11-0bd0-445d-8e44-388ceaf9418a-kube-api-access-br52p\") pod \"026dbd11-0bd0-445d-8e44-388ceaf9418a\" (UID: \"026dbd11-0bd0-445d-8e44-388ceaf9418a\") "
Oct 30 20:11:41.443523 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:41.439756    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/026dbd11-0bd0-445d-8e44-388ceaf9418a-plugins-config-volume" pod "026dbd11-0bd0-445d-8e44-388ceaf9418a" (UID: "026dbd11-0bd0-445d-8e44-388ceaf9418a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:41.443523 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:41.442864    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026dbd11-0bd0-445d-8e44-388ceaf9418a-kube-api-access-br52p" pod "026dbd11-0bd0-445d-8e44-388ceaf9418a" (UID: "026dbd11-0bd0-445d-8e44-388ceaf9418a"). InnerVolumeSpecName "kube-api-access-br52p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:41.478135 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:41.478106    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc"
Oct 30 20:11:41.540494 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:41.540432    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc1532ea-e75f-4f6e-9791-8e95bcf51cf8-kube-api-access-j8j84\" (UniqueName: \"kubernetes.io/projected/fc1532ea-e75f-4f6e-9791-8e95bcf51cf8-kube-api-access-j8j84\") pod \"fc1532ea-e75f-4f6e-9791-8e95bcf51cf8\" (UID: \"fc1532ea-e75f-4f6e-9791-8e95bcf51cf8\") "
Oct 30 20:11:41.540658 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:41.540607    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc1532ea-e75f-4f6e-9791-8e95bcf51cf8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1532ea-e75f-4f6e-9791-8e95bcf51cf8-plugins-config-volume\") pod \"fc1532ea-e75f-4f6e-9791-8e95bcf51cf8\" (UID: \"fc1532ea-e75f-4f6e-9791-8e95bcf51cf8\") "
Oct 30 20:11:41.541005 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:41.540977    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc1532ea-e75f-4f6e-9791-8e95bcf51cf8-plugins-config-volume" pod "fc1532ea-e75f-4f6e-9791-8e95bcf51cf8" (UID: "fc1532ea-e75f-4f6e-9791-8e95bcf51cf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:41.541413 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:41.541392    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1532ea-e75f-4f6e-9791-8e95bcf51cf8-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:41.541514 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:41.541414    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/026dbd11-0bd0-445d-8e44-388ceaf9418a-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:41.541514 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:41.541425    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br52p\" (UniqueName: \"kubernetes.io/projected/026dbd11-0bd0-445d-8e44-388ceaf9418a-kube-api-access-br52p\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:41.542955 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:41.542924    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc1532ea-e75f-4f6e-9791-8e95bcf51cf8-kube-api-access-j8j84" pod "fc1532ea-e75f-4f6e-9791-8e95bcf51cf8" (UID: "fc1532ea-e75f-4f6e-9791-8e95bcf51cf8"). InnerVolumeSpecName "kube-api-access-j8j84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:41.644960 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:41.644840    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8j84\" (UniqueName: \"kubernetes.io/projected/fc1532ea-e75f-4f6e-9791-8e95bcf51cf8-kube-api-access-j8j84\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:41.979873 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:41.979787    5958 generic.go:356] "Generic (PLEG): container finished" podID="fc1532ea-e75f-4f6e-9791-8e95bcf51cf8" containerID="2c5caeafd700ae0de212bbc5e4bf1fae2b125a4175e26e3621378f69525e49f8" exitCode=0
Oct 30 20:11:41.979873 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:41.979852    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc"
Oct 30 20:11:41.980114 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:41.979877    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc" event={"ID":"fc1532ea-e75f-4f6e-9791-8e95bcf51cf8","Type":"ContainerDied","Data":"2c5caeafd700ae0de212bbc5e4bf1fae2b125a4175e26e3621378f69525e49f8"}
Oct 30 20:11:41.980114 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:41.979944    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc" event={"ID":"fc1532ea-e75f-4f6e-9791-8e95bcf51cf8","Type":"ContainerDied","Data":"837e6705e86a8727fa86c14a0c16bd02c9eb46ae02599338b7b1494524eb4d23"}
Oct 30 20:11:41.980114 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:41.979973    5958 scope.go:122] "RemoveContainer" containerID="2c5caeafd700ae0de212bbc5e4bf1fae2b125a4175e26e3621378f69525e49f8"
Oct 30 20:11:41.990475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:41.985622    5958 generic.go:356] "Generic (PLEG): container finished" podID="026dbd11-0bd0-445d-8e44-388ceaf9418a" containerID="5c382a5097345aad32763d58eaa5fdded497870bbd9942ef8f8ebf947ef3ffa2" exitCode=0
Oct 30 20:11:41.990475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:41.985777    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv" event={"ID":"026dbd11-0bd0-445d-8e44-388ceaf9418a","Type":"ContainerDied","Data":"5c382a5097345aad32763d58eaa5fdded497870bbd9942ef8f8ebf947ef3ffa2"}
Oct 30 20:11:41.990475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:41.985839    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv" event={"ID":"026dbd11-0bd0-445d-8e44-388ceaf9418a","Type":"ContainerDied","Data":"a343a5c04c2a71d3708989ea9905f64ea75c718c982a5c73704a42743ecd87de"}
Oct 30 20:11:41.990475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:41.985942    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv"
Oct 30 20:11:42.055480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.055014    5958 scope.go:122] "RemoveContainer" containerID="2c5caeafd700ae0de212bbc5e4bf1fae2b125a4175e26e3621378f69525e49f8"
Oct 30 20:11:42.056245 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:11:42.056213    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c5caeafd700ae0de212bbc5e4bf1fae2b125a4175e26e3621378f69525e49f8\": container with ID starting with 2c5caeafd700ae0de212bbc5e4bf1fae2b125a4175e26e3621378f69525e49f8 not found: ID does not exist" containerID="2c5caeafd700ae0de212bbc5e4bf1fae2b125a4175e26e3621378f69525e49f8"
Oct 30 20:11:42.056547 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.056257    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c5caeafd700ae0de212bbc5e4bf1fae2b125a4175e26e3621378f69525e49f8"} err="failed to get container status \"2c5caeafd700ae0de212bbc5e4bf1fae2b125a4175e26e3621378f69525e49f8\": rpc error: code = NotFound desc = could not find container \"2c5caeafd700ae0de212bbc5e4bf1fae2b125a4175e26e3621378f69525e49f8\": container with ID starting with 2c5caeafd700ae0de212bbc5e4bf1fae2b125a4175e26e3621378f69525e49f8 not found: ID does not exist"
Oct 30 20:11:42.056547 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.056283    5958 scope.go:122] "RemoveContainer" containerID="5c382a5097345aad32763d58eaa5fdded497870bbd9942ef8f8ebf947ef3ffa2"
Oct 30 20:11:42.089764 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.089730    5958 scope.go:122] "RemoveContainer" containerID="5c382a5097345aad32763d58eaa5fdded497870bbd9942ef8f8ebf947ef3ffa2"
Oct 30 20:11:42.090218 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:11:42.090190    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c382a5097345aad32763d58eaa5fdded497870bbd9942ef8f8ebf947ef3ffa2\": container with ID starting with 5c382a5097345aad32763d58eaa5fdded497870bbd9942ef8f8ebf947ef3ffa2 not found: ID does not exist" containerID="5c382a5097345aad32763d58eaa5fdded497870bbd9942ef8f8ebf947ef3ffa2"
Oct 30 20:11:42.090278 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.090233    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c382a5097345aad32763d58eaa5fdded497870bbd9942ef8f8ebf947ef3ffa2"} err="failed to get container status \"5c382a5097345aad32763d58eaa5fdded497870bbd9942ef8f8ebf947ef3ffa2\": rpc error: code = NotFound desc = could not find container \"5c382a5097345aad32763d58eaa5fdded497870bbd9942ef8f8ebf947ef3ffa2\": container with ID starting with 5c382a5097345aad32763d58eaa5fdded497870bbd9942ef8f8ebf947ef3ffa2 not found: ID does not exist"
Oct 30 20:11:42.210289 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.210251    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc"]
Oct 30 20:11:42.217243 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.217202    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-80e4d90e-587fbb6c47-mkmkc"]
Oct 30 20:11:42.303122 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.303042    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn"]
Oct 30 20:11:42.303870 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.303851    5958 cpu_manager.go:414] "containerMap: removing container" podUID="fc1532ea-e75f-4f6e-9791-8e95bcf51cf8" containerName="payload-pre-processing"
Oct 30 20:11:42.303870 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.303868    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc1532ea-e75f-4f6e-9791-8e95bcf51cf8" containerName="payload-pre-processing"
Oct 30 20:11:42.304018 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.303895    5958 cpu_manager.go:414] "containerMap: removing container" podUID="026dbd11-0bd0-445d-8e44-388ceaf9418a" containerName="payload-processing"
Oct 30 20:11:42.304018 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.303901    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026dbd11-0bd0-445d-8e44-388ceaf9418a" containerName="payload-processing"
Oct 30 20:11:42.304018 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.303941    5958 cpu_manager.go:414] "containerMap: removing container" podUID="6530268f-b6ee-4315-ab68-affa9a4057f9" containerName="payload-pre-processing"
Oct 30 20:11:42.304018 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.303946    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6530268f-b6ee-4315-ab68-affa9a4057f9" containerName="payload-pre-processing"
Oct 30 20:11:42.304018 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.303962    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4" containerName="payload-processing"
Oct 30 20:11:42.304018 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.303966    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4" containerName="payload-processing"
Oct 30 20:11:42.304187 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.304062    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="026dbd11-0bd0-445d-8e44-388ceaf9418a" containerName="payload-processing"
Oct 30 20:11:42.304187 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.304080    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="6530268f-b6ee-4315-ab68-affa9a4057f9" containerName="payload-pre-processing"
Oct 30 20:11:42.304187 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.304089    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceb9bf2d-55cf-470b-a640-8c50c6d2d8b4" containerName="payload-processing"
Oct 30 20:11:42.304187 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.304096    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc1532ea-e75f-4f6e-9791-8e95bcf51cf8" containerName="payload-pre-processing"
Oct 30 20:11:42.307615 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.307591    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn"
Oct 30 20:11:42.333502 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.327793    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv"]
Oct 30 20:11:42.336700 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.334868    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-80e4d90e-8584445c74-xjszv"]
Oct 30 20:11:42.361484 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.360666    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwhrs\" (UniqueName: \"kubernetes.io/projected/c35c95a6-d115-43cf-a332-19dc91c7bdd4-kube-api-access-qwhrs\") pod \"payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn\" (UID: \"c35c95a6-d115-43cf-a332-19dc91c7bdd4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn"
Oct 30 20:11:42.361484 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.360796    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35c95a6-d115-43cf-a332-19dc91c7bdd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn\" (UID: \"c35c95a6-d115-43cf-a332-19dc91c7bdd4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn"
Oct 30 20:11:42.372479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.369780    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-7961ed-dockercfg-7ngmj\""
Oct 30 20:11:42.372479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.369892    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-7961ed\""
Oct 30 20:11:42.397544 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.395788    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn"]
Oct 30 20:11:42.464228 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.464175    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwhrs\" (UniqueName: \"kubernetes.io/projected/c35c95a6-d115-43cf-a332-19dc91c7bdd4-kube-api-access-qwhrs\") pod \"payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn\" (UID: \"c35c95a6-d115-43cf-a332-19dc91c7bdd4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn"
Oct 30 20:11:42.464430 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.464346    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35c95a6-d115-43cf-a332-19dc91c7bdd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn\" (UID: \"c35c95a6-d115-43cf-a332-19dc91c7bdd4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn"
Oct 30 20:11:42.467218 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.467184    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35c95a6-d115-43cf-a332-19dc91c7bdd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn\" (UID: \"c35c95a6-d115-43cf-a332-19dc91c7bdd4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn"
Oct 30 20:11:42.530522 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.530470    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwhrs\" (UniqueName: \"kubernetes.io/projected/c35c95a6-d115-43cf-a332-19dc91c7bdd4-kube-api-access-qwhrs\") pod \"payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn\" (UID: \"c35c95a6-d115-43cf-a332-19dc91c7bdd4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn"
Oct 30 20:11:42.650472 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:42.650018    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn"
Oct 30 20:11:43.052259 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:43.052092    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn"]
Oct 30 20:11:43.401322 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:43.401288    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="026dbd11-0bd0-445d-8e44-388ceaf9418a" path="/var/lib/kubelet/pods/026dbd11-0bd0-445d-8e44-388ceaf9418a/volumes"
Oct 30 20:11:43.401848 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:43.401827    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc1532ea-e75f-4f6e-9791-8e95bcf51cf8" path="/var/lib/kubelet/pods/fc1532ea-e75f-4f6e-9791-8e95bcf51cf8/volumes"
Oct 30 20:11:44.023722 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:44.023678    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn" event={"ID":"c35c95a6-d115-43cf-a332-19dc91c7bdd4","Type":"ContainerStarted","Data":"2226ed7b59e41b7dc00d12982a8192b3e8e556a3ffcdedaf4a3997784de659af"}
Oct 30 20:11:44.023920 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:44.023747    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn" event={"ID":"c35c95a6-d115-43cf-a332-19dc91c7bdd4","Type":"ContainerStarted","Data":"632a52d96ab7d12ca467bf3d2636a77afac1b741e84658d64b30b9155afe01f2"}
Oct 30 20:11:44.023920 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:44.023773    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn"
Oct 30 20:11:44.143415 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:44.143360    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn" podStartSLOduration=2.143346651 podStartE2EDuration="2.143346651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:44.140143771 +0000 UTC m=+1742.872359162" watchObservedRunningTime="2026-10-30 20:11:44.143346651 +0000 UTC m=+1742.875562041"
Oct 30 20:11:49.100607 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:49.100569    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk"]
Oct 30 20:11:54.102482 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:54.101517    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk" podUID="4017f735-ee90-40c3-ba71-4e57ddf03681" containerName="payload-processing" containerID="cri-o://3f7e979e0577dcc61d3169c07f1350e9c4324bcbf6f7ce524078632406a4c77b" gracePeriod=25
Oct 30 20:11:54.709300 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:54.709264    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk"
Oct 30 20:11:54.833103 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:54.833034    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4017f735-ee90-40c3-ba71-4e57ddf03681-kube-api-access-pr5ds\" (UniqueName: \"kubernetes.io/projected/4017f735-ee90-40c3-ba71-4e57ddf03681-kube-api-access-pr5ds\") pod \"4017f735-ee90-40c3-ba71-4e57ddf03681\" (UID: \"4017f735-ee90-40c3-ba71-4e57ddf03681\") "
Oct 30 20:11:54.833599 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:54.833573    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4017f735-ee90-40c3-ba71-4e57ddf03681-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4017f735-ee90-40c3-ba71-4e57ddf03681-plugins-config-volume\") pod \"4017f735-ee90-40c3-ba71-4e57ddf03681\" (UID: \"4017f735-ee90-40c3-ba71-4e57ddf03681\") "
Oct 30 20:11:54.834074 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:54.834044    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4017f735-ee90-40c3-ba71-4e57ddf03681-plugins-config-volume" pod "4017f735-ee90-40c3-ba71-4e57ddf03681" (UID: "4017f735-ee90-40c3-ba71-4e57ddf03681"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:54.834589 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:54.834564    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4017f735-ee90-40c3-ba71-4e57ddf03681-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:54.848477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:54.835547    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4017f735-ee90-40c3-ba71-4e57ddf03681-kube-api-access-pr5ds" pod "4017f735-ee90-40c3-ba71-4e57ddf03681" (UID: "4017f735-ee90-40c3-ba71-4e57ddf03681"). InnerVolumeSpecName "kube-api-access-pr5ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:54.936736 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:54.936632    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr5ds\" (UniqueName: \"kubernetes.io/projected/4017f735-ee90-40c3-ba71-4e57ddf03681-kube-api-access-pr5ds\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:55.038951 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:55.038896    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn"
Oct 30 20:11:55.131093 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:55.131054    5958 generic.go:356] "Generic (PLEG): container finished" podID="4017f735-ee90-40c3-ba71-4e57ddf03681" containerID="3f7e979e0577dcc61d3169c07f1350e9c4324bcbf6f7ce524078632406a4c77b" exitCode=0
Oct 30 20:11:55.131523 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:55.131106    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk"
Oct 30 20:11:55.131523 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:55.131138    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk" event={"ID":"4017f735-ee90-40c3-ba71-4e57ddf03681","Type":"ContainerDied","Data":"3f7e979e0577dcc61d3169c07f1350e9c4324bcbf6f7ce524078632406a4c77b"}
Oct 30 20:11:55.131523 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:55.131195    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk" event={"ID":"4017f735-ee90-40c3-ba71-4e57ddf03681","Type":"ContainerDied","Data":"82b6925367be3f8b3dc220cc8f3fa81e1e4a3aa52046e6d2009911d19a474005"}
Oct 30 20:11:55.131523 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:55.131223    5958 scope.go:122] "RemoveContainer" containerID="3f7e979e0577dcc61d3169c07f1350e9c4324bcbf6f7ce524078632406a4c77b"
Oct 30 20:11:55.143595 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:55.143569    5958 scope.go:122] "RemoveContainer" containerID="3f7e979e0577dcc61d3169c07f1350e9c4324bcbf6f7ce524078632406a4c77b"
Oct 30 20:11:55.143954 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:11:55.143933    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f7e979e0577dcc61d3169c07f1350e9c4324bcbf6f7ce524078632406a4c77b\": container with ID starting with 3f7e979e0577dcc61d3169c07f1350e9c4324bcbf6f7ce524078632406a4c77b not found: ID does not exist" containerID="3f7e979e0577dcc61d3169c07f1350e9c4324bcbf6f7ce524078632406a4c77b"
Oct 30 20:11:55.144036 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:55.143966    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7e979e0577dcc61d3169c07f1350e9c4324bcbf6f7ce524078632406a4c77b"} err="failed to get container status \"3f7e979e0577dcc61d3169c07f1350e9c4324bcbf6f7ce524078632406a4c77b\": rpc error: code = NotFound desc = could not find container \"3f7e979e0577dcc61d3169c07f1350e9c4324bcbf6f7ce524078632406a4c77b\": container with ID starting with 3f7e979e0577dcc61d3169c07f1350e9c4324bcbf6f7ce524078632406a4c77b not found: ID does not exist"
Oct 30 20:11:55.273821 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:55.273738    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk"]
Oct 30 20:11:55.289703 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:55.289663    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d7ca5342-559c6c844-dxhjk"]
Oct 30 20:11:55.401720 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:55.401685    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4017f735-ee90-40c3-ba71-4e57ddf03681" path="/var/lib/kubelet/pods/4017f735-ee90-40c3-ba71-4e57ddf03681/volumes"
Oct 30 20:11:59.055358 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:59.055323    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55e619f8-27d7k"]
Oct 30 20:11:59.105376 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:59.105307    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55e619f8-27d7k"]
Oct 30 20:11:59.401100 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:11:59.401057    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ada3b129-dc27-4d16-8707-e28d00d1cc2d" path="/var/lib/kubelet/pods/ada3b129-dc27-4d16-8707-e28d00d1cc2d/volumes"
Oct 30 20:12:00.653726 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.653691    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"]
Oct 30 20:12:00.654514 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.654492    5958 cpu_manager.go:414] "containerMap: removing container" podUID="4017f735-ee90-40c3-ba71-4e57ddf03681" containerName="payload-processing"
Oct 30 20:12:00.654592 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.654516    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4017f735-ee90-40c3-ba71-4e57ddf03681" containerName="payload-processing"
Oct 30 20:12:00.654691 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.654681    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="4017f735-ee90-40c3-ba71-4e57ddf03681" containerName="payload-processing"
Oct 30 20:12:00.658602 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.658582    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:12:00.711757 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.711730    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 20:12:00.711882 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.711771    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 20:12:00.711882 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.711772    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 20:12:00.761345 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.761323    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jf4m5\""
Oct 30 20:12:00.768509 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.768484    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"]
Oct 30 20:12:00.826635 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.826609    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f8392a2-9219-4e55-88a8-955e72c0f98c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkjvc\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:12:00.826804 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.826788    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkjvc\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:12:00.826851 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.826812    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkjvc\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:12:00.826901 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.826856    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkjvc\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:12:00.827045 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.827024    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkjvc\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:12:00.827092 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.827067    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8392a2-9219-4e55-88a8-955e72c0f98c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkjvc\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:12:00.827199 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.827186    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnjcn\" (UniqueName: \"kubernetes.io/projected/9f8392a2-9219-4e55-88a8-955e72c0f98c-kube-api-access-rnjcn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkjvc\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:12:00.928636 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.928570    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkjvc\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:12:00.928636 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.928614    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8392a2-9219-4e55-88a8-955e72c0f98c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkjvc\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:12:00.928768 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.928662    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnjcn\" (UniqueName: \"kubernetes.io/projected/9f8392a2-9219-4e55-88a8-955e72c0f98c-kube-api-access-rnjcn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkjvc\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:12:00.928768 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.928697    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f8392a2-9219-4e55-88a8-955e72c0f98c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkjvc\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:12:00.928838 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.928805    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkjvc\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:12:00.928838 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.928831    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkjvc\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:12:00.928914 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.928887    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkjvc\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:12:00.929602 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.929572    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkjvc\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:12:00.931084 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.931062    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8392a2-9219-4e55-88a8-955e72c0f98c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkjvc\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:12:00.931221 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.931203    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkjvc\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:12:00.931713 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.931693    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f8392a2-9219-4e55-88a8-955e72c0f98c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkjvc\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:12:00.931812 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.931724    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkjvc\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:12:00.990501 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.990481    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkjvc\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:12:00.990699 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:00.990679    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnjcn\" (UniqueName: \"kubernetes.io/projected/9f8392a2-9219-4e55-88a8-955e72c0f98c-kube-api-access-rnjcn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bkjvc\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:12:01.272130 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:01.272067    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:12:01.707034 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:01.707006    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"]
Oct 30 20:12:01.707597 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:12:01.707575    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f8392a2_9219_4e55_88a8_955e72c0f98c.slice/crio-8ea7b8252c165b6c2b511b12e03f4c751133b915775caec49fd66d1dd34b8051 WatchSource:0}: Error finding container 8ea7b8252c165b6c2b511b12e03f4c751133b915775caec49fd66d1dd34b8051: Status 404 returned error can't find the container with id 8ea7b8252c165b6c2b511b12e03f4c751133b915775caec49fd66d1dd34b8051
Oct 30 20:12:02.191860 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:02.191826    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc" event={"ID":"9f8392a2-9219-4e55-88a8-955e72c0f98c","Type":"ContainerStarted","Data":"8ea7b8252c165b6c2b511b12e03f4c751133b915775caec49fd66d1dd34b8051"}
Oct 30 20:12:06.607942 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:12:06.607898    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:12:10.271092 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:10.271053    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:12:10.271377 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:10.271123    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:12:11.358629 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:11.358582    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc" event={"ID":"9f8392a2-9219-4e55-88a8-955e72c0f98c","Type":"ContainerStarted","Data":"4295f80209bbd28b13aed32dfd2c9e4e1b045c65b8fc369f964f5c71c5b87fa6"}
Oct 30 20:12:11.359094 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:11.358822    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:12:11.360555 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:11.360521    5958 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bkjvc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:12:11.360669 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:11.360587    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc" podUID="9f8392a2-9219-4e55-88a8-955e72c0f98c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:12:11.574069 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:11.574005    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc" podStartSLOduration=3.014305702 podStartE2EDuration="11.57398712s" totalImagesPullingTime="8.559681418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:11.530977514 +0000 UTC m=+1770.263192902" watchObservedRunningTime="2026-10-30 20:12:11.57398712 +0000 UTC m=+1770.306202512"
Oct 30 20:12:12.367047 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:12.367020    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:12:15.316511 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:15.316477    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525"]
Oct 30 20:12:15.316928 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:15.316692    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525" podUID="7b8f2b7f-2e80-4fde-ac00-ee68816804be" containerName="maas-api" containerID="cri-o://1a1b093d8f5968e785fabd63cd39a88bb85893d7317276073bf0a550c45b4f2e" gracePeriod=30
Oct 30 20:12:15.723430 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:15.723405    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525"
Oct 30 20:12:15.803065 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:15.803020    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b8f2b7f-2e80-4fde-ac00-ee68816804be-kube-api-access-zrqjk\" (UniqueName: \"kubernetes.io/projected/7b8f2b7f-2e80-4fde-ac00-ee68816804be-kube-api-access-zrqjk\") pod \"7b8f2b7f-2e80-4fde-ac00-ee68816804be\" (UID: \"7b8f2b7f-2e80-4fde-ac00-ee68816804be\") "
Oct 30 20:12:15.803387 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:15.803364    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b8f2b7f-2e80-4fde-ac00-ee68816804be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b8f2b7f-2e80-4fde-ac00-ee68816804be-maas-api-tls\") pod \"7b8f2b7f-2e80-4fde-ac00-ee68816804be\" (UID: \"7b8f2b7f-2e80-4fde-ac00-ee68816804be\") "
Oct 30 20:12:15.803531 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:15.803426    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b8f2b7f-2e80-4fde-ac00-ee68816804be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b8f2b7f-2e80-4fde-ac00-ee68816804be-openshift-service-ca\") pod \"7b8f2b7f-2e80-4fde-ac00-ee68816804be\" (UID: \"7b8f2b7f-2e80-4fde-ac00-ee68816804be\") "
Oct 30 20:12:15.803610 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:15.803591    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b8f2b7f-2e80-4fde-ac00-ee68816804be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b8f2b7f-2e80-4fde-ac00-ee68816804be-metrics-certs\") pod \"7b8f2b7f-2e80-4fde-ac00-ee68816804be\" (UID: \"7b8f2b7f-2e80-4fde-ac00-ee68816804be\") "
Oct 30 20:12:15.805084 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:15.805058    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b8f2b7f-2e80-4fde-ac00-ee68816804be-openshift-service-ca" pod "7b8f2b7f-2e80-4fde-ac00-ee68816804be" (UID: "7b8f2b7f-2e80-4fde-ac00-ee68816804be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:15.806081 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:15.806047    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b8f2b7f-2e80-4fde-ac00-ee68816804be-kube-api-access-zrqjk" pod "7b8f2b7f-2e80-4fde-ac00-ee68816804be" (UID: "7b8f2b7f-2e80-4fde-ac00-ee68816804be"). InnerVolumeSpecName "kube-api-access-zrqjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:15.806282 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:15.806255    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b8f2b7f-2e80-4fde-ac00-ee68816804be-maas-api-tls" pod "7b8f2b7f-2e80-4fde-ac00-ee68816804be" (UID: "7b8f2b7f-2e80-4fde-ac00-ee68816804be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:15.807127 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:15.807103    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b8f2b7f-2e80-4fde-ac00-ee68816804be-metrics-certs" pod "7b8f2b7f-2e80-4fde-ac00-ee68816804be" (UID: "7b8f2b7f-2e80-4fde-ac00-ee68816804be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:15.906582 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:15.906493    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b8f2b7f-2e80-4fde-ac00-ee68816804be-metrics-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:15.906582 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:15.906525    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrqjk\" (UniqueName: \"kubernetes.io/projected/7b8f2b7f-2e80-4fde-ac00-ee68816804be-kube-api-access-zrqjk\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:15.906582 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:15.906538    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b8f2b7f-2e80-4fde-ac00-ee68816804be-maas-api-tls\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:15.906582 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:15.906572    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b8f2b7f-2e80-4fde-ac00-ee68816804be-openshift-service-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:16.401830 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:16.401791    5958 generic.go:356] "Generic (PLEG): container finished" podID="7b8f2b7f-2e80-4fde-ac00-ee68816804be" containerID="1a1b093d8f5968e785fabd63cd39a88bb85893d7317276073bf0a550c45b4f2e" exitCode=0
Oct 30 20:12:16.402330 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:16.401888    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525"
Oct 30 20:12:16.402330 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:16.401932    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525" event={"ID":"7b8f2b7f-2e80-4fde-ac00-ee68816804be","Type":"ContainerDied","Data":"1a1b093d8f5968e785fabd63cd39a88bb85893d7317276073bf0a550c45b4f2e"}
Oct 30 20:12:16.402330 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:16.401999    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525" event={"ID":"7b8f2b7f-2e80-4fde-ac00-ee68816804be","Type":"ContainerDied","Data":"84c10798cac4578138bf84c458b9f6e3726e171aaa0d66fe2f431e3faff3671b"}
Oct 30 20:12:16.402330 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:16.402029    5958 scope.go:122] "RemoveContainer" containerID="1a1b093d8f5968e785fabd63cd39a88bb85893d7317276073bf0a550c45b4f2e"
Oct 30 20:12:16.415515 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:16.415494    5958 scope.go:122] "RemoveContainer" containerID="1a1b093d8f5968e785fabd63cd39a88bb85893d7317276073bf0a550c45b4f2e"
Oct 30 20:12:16.415809 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:12:16.415788    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a1b093d8f5968e785fabd63cd39a88bb85893d7317276073bf0a550c45b4f2e\": container with ID starting with 1a1b093d8f5968e785fabd63cd39a88bb85893d7317276073bf0a550c45b4f2e not found: ID does not exist" containerID="1a1b093d8f5968e785fabd63cd39a88bb85893d7317276073bf0a550c45b4f2e"
Oct 30 20:12:16.415869 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:16.415821    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a1b093d8f5968e785fabd63cd39a88bb85893d7317276073bf0a550c45b4f2e"} err="failed to get container status \"1a1b093d8f5968e785fabd63cd39a88bb85893d7317276073bf0a550c45b4f2e\": rpc error: code = NotFound desc = could not find container \"1a1b093d8f5968e785fabd63cd39a88bb85893d7317276073bf0a550c45b4f2e\": container with ID starting with 1a1b093d8f5968e785fabd63cd39a88bb85893d7317276073bf0a550c45b4f2e not found: ID does not exist"
Oct 30 20:12:16.574802 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:16.574773    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525"]
Oct 30 20:12:16.624278 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:16.624245    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-27915880-6489d6dc7-sb525"]
Oct 30 20:12:17.405585 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:17.405535    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b8f2b7f-2e80-4fde-ac00-ee68816804be" path="/var/lib/kubelet/pods/7b8f2b7f-2e80-4fde-ac00-ee68816804be/volumes"
Oct 30 20:12:18.206145 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:18.206108    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"]
Oct 30 20:12:18.206723 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:18.206672    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6" podUID="33bf491a-644a-4a04-b8d5-5ca6842ff3e6" containerName="istio-proxy" containerID="cri-o://c863b30ecd2a6d8b69f96b0024dee9c2bd8173efe8d428142cf68d582ca329d6" gracePeriod=30
Oct 30 20:12:20.052075 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:20.052042    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"]
Oct 30 20:12:20.052480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:20.052303    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c" podUID="ede036c9-86b3-4d4b-95b0-88570503f1c2" containerName="istio-proxy" containerID="cri-o://ed6100f1fb6f45d55fe247130b951bd59c0285dbc9c9b308e742ee37d90997c6" gracePeriod=30
Oct 30 20:12:22.283782 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:22.283740    5958 patch_prober.go:29] interesting pod/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:12:22.284158 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:22.283797    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c" podUID="ede036c9-86b3-4d4b-95b0-88570503f1c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:12:23.203907 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.203862    5958 patch_prober.go:29] interesting pod/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:12:23.204080 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.203928    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6" podUID="33bf491a-644a-4a04-b8d5-5ca6842ff3e6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:12:23.469699 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.469621    5958 generic.go:356] "Generic (PLEG): container finished" podID="33bf491a-644a-4a04-b8d5-5ca6842ff3e6" containerID="c863b30ecd2a6d8b69f96b0024dee9c2bd8173efe8d428142cf68d582ca329d6" exitCode=0
Oct 30 20:12:23.469699 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.469681    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6" event={"ID":"33bf491a-644a-4a04-b8d5-5ca6842ff3e6","Type":"ContainerDied","Data":"c863b30ecd2a6d8b69f96b0024dee9c2bd8173efe8d428142cf68d582ca329d6"}
Oct 30 20:12:23.470203 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.469762    5958 scope.go:122] "RemoveContainer" containerID="5041afe3a7604006b478cc2e63cf910ce0291f8f47d04fc298fa48ad9f16383b"
Oct 30 20:12:23.508362 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.508341    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:12:23.648096 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.648065    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istiod-ca-cert\") pod \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") "
Oct 30 20:12:23.648237 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.648109    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-token\" (UniqueName: \"kubernetes.io/projected/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-token\") pod \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") "
Oct 30 20:12:23.648237 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.648148    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-workload-certs\") pod \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") "
Oct 30 20:12:23.648358 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.648332    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-data\") pod \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") "
Oct 30 20:12:23.648530 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.648503    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-workload-certs" pod "33bf491a-644a-4a04-b8d5-5ca6842ff3e6" (UID: "33bf491a-644a-4a04-b8d5-5ca6842ff3e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:23.648530 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.648510    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istiod-ca-cert" pod "33bf491a-644a-4a04-b8d5-5ca6842ff3e6" (UID: "33bf491a-644a-4a04-b8d5-5ca6842ff3e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:23.648685 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.648527    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-podinfo\") pod \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") "
Oct 30 20:12:23.648685 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.648596    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-workload-socket\") pod \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") "
Oct 30 20:12:23.648685 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.648609    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-data" pod "33bf491a-644a-4a04-b8d5-5ca6842ff3e6" (UID: "33bf491a-644a-4a04-b8d5-5ca6842ff3e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:23.648685 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.648648    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-kube-api-access-tbp4z\" (UniqueName: \"kubernetes.io/projected/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-kube-api-access-tbp4z\") pod \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") "
Oct 30 20:12:23.648854 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.648778    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-credential-socket\") pod \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") "
Oct 30 20:12:23.648854 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.648814    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-envoy\") pod \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\" (UID: \"33bf491a-644a-4a04-b8d5-5ca6842ff3e6\") "
Oct 30 20:12:23.648960 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.648872    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-workload-socket" pod "33bf491a-644a-4a04-b8d5-5ca6842ff3e6" (UID: "33bf491a-644a-4a04-b8d5-5ca6842ff3e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:23.649219 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.649195    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-credential-socket" pod "33bf491a-644a-4a04-b8d5-5ca6842ff3e6" (UID: "33bf491a-644a-4a04-b8d5-5ca6842ff3e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:23.649730 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.649705    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:23.649850 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.649737    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:23.649850 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.649753    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:23.649850 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.649765    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:23.649850 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.649778    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:23.650982 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.650957    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-token" pod "33bf491a-644a-4a04-b8d5-5ca6842ff3e6" (UID: "33bf491a-644a-4a04-b8d5-5ca6842ff3e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:23.651304 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.651275    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-podinfo" pod "33bf491a-644a-4a04-b8d5-5ca6842ff3e6" (UID: "33bf491a-644a-4a04-b8d5-5ca6842ff3e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:12:23.651430 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.651380    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-envoy" pod "33bf491a-644a-4a04-b8d5-5ca6842ff3e6" (UID: "33bf491a-644a-4a04-b8d5-5ca6842ff3e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:23.651576 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.651552    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-kube-api-access-tbp4z" pod "33bf491a-644a-4a04-b8d5-5ca6842ff3e6" (UID: "33bf491a-644a-4a04-b8d5-5ca6842ff3e6"). InnerVolumeSpecName "kube-api-access-tbp4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:23.751600 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.751527    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbp4z\" (UniqueName: \"kubernetes.io/projected/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-kube-api-access-tbp4z\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:23.751600 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.751555    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:23.751600 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.751567    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:23.751600 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:23.751575    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33bf491a-644a-4a04-b8d5-5ca6842ff3e6-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:24.479396 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:24.479366    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"
Oct 30 20:12:24.479881 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:24.479405    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6" event={"ID":"33bf491a-644a-4a04-b8d5-5ca6842ff3e6","Type":"ContainerDied","Data":"11d0e856d5f2f4298ecad1ff6049a94d01f6729f68e80f5683ece27871ffd498"}
Oct 30 20:12:24.479881 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:24.479475    5958 scope.go:122] "RemoveContainer" containerID="c863b30ecd2a6d8b69f96b0024dee9c2bd8173efe8d428142cf68d582ca329d6"
Oct 30 20:12:24.766265 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:24.766177    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"]
Oct 30 20:12:24.770760 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:24.770735    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-27915880-gw-openshift-default-55649b4b97-rmrm6"]
Oct 30 20:12:25.139769 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.139741    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn"]
Oct 30 20:12:25.139958 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.139937    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn" podUID="90ae88c8-fb47-43da-8804-c10a62cb2179" containerName="maas-api" containerID="cri-o://c25fff084a384832a3c9796073966dd8b0a83b2931e2117dd89d6b76ff9e1e66" gracePeriod=30
Oct 30 20:12:25.408724 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.408697    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33bf491a-644a-4a04-b8d5-5ca6842ff3e6" path="/var/lib/kubelet/pods/33bf491a-644a-4a04-b8d5-5ca6842ff3e6/volumes"
Oct 30 20:12:25.495371 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.495341    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:12:25.500857 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.500819    5958 generic.go:356] "Generic (PLEG): container finished" podID="90ae88c8-fb47-43da-8804-c10a62cb2179" containerID="c25fff084a384832a3c9796073966dd8b0a83b2931e2117dd89d6b76ff9e1e66" exitCode=0
Oct 30 20:12:25.501078 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.500903    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn" event={"ID":"90ae88c8-fb47-43da-8804-c10a62cb2179","Type":"ContainerDied","Data":"c25fff084a384832a3c9796073966dd8b0a83b2931e2117dd89d6b76ff9e1e66"}
Oct 30 20:12:25.503276 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.503250    5958 generic.go:356] "Generic (PLEG): container finished" podID="ede036c9-86b3-4d4b-95b0-88570503f1c2" containerID="ed6100f1fb6f45d55fe247130b951bd59c0285dbc9c9b308e742ee37d90997c6" exitCode=0
Oct 30 20:12:25.503401 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.503294    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"
Oct 30 20:12:25.503401 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.503348    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c" event={"ID":"ede036c9-86b3-4d4b-95b0-88570503f1c2","Type":"ContainerDied","Data":"ed6100f1fb6f45d55fe247130b951bd59c0285dbc9c9b308e742ee37d90997c6"}
Oct 30 20:12:25.503535 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.503405    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c" event={"ID":"ede036c9-86b3-4d4b-95b0-88570503f1c2","Type":"ContainerDied","Data":"06324203575c67e784afc8e85b4466612cec1983e898e9579eb8fb40eedde7a0"}
Oct 30 20:12:25.503535 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.503435    5958 scope.go:122] "RemoveContainer" containerID="ed6100f1fb6f45d55fe247130b951bd59c0285dbc9c9b308e742ee37d90997c6"
Oct 30 20:12:25.524868 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.524852    5958 scope.go:122] "RemoveContainer" containerID="f5f7b43196475efd0db9db8770deabe65b8a5276355d90df18320252ede3dbb2"
Oct 30 20:12:25.527213 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.527193    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn"
Oct 30 20:12:25.536225 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.536196    5958 scope.go:122] "RemoveContainer" containerID="ed6100f1fb6f45d55fe247130b951bd59c0285dbc9c9b308e742ee37d90997c6"
Oct 30 20:12:25.536588 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:12:25.536558    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed6100f1fb6f45d55fe247130b951bd59c0285dbc9c9b308e742ee37d90997c6\": container with ID starting with ed6100f1fb6f45d55fe247130b951bd59c0285dbc9c9b308e742ee37d90997c6 not found: ID does not exist" containerID="ed6100f1fb6f45d55fe247130b951bd59c0285dbc9c9b308e742ee37d90997c6"
Oct 30 20:12:25.536701 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.536596    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed6100f1fb6f45d55fe247130b951bd59c0285dbc9c9b308e742ee37d90997c6"} err="failed to get container status \"ed6100f1fb6f45d55fe247130b951bd59c0285dbc9c9b308e742ee37d90997c6\": rpc error: code = NotFound desc = could not find container \"ed6100f1fb6f45d55fe247130b951bd59c0285dbc9c9b308e742ee37d90997c6\": container with ID starting with ed6100f1fb6f45d55fe247130b951bd59c0285dbc9c9b308e742ee37d90997c6 not found: ID does not exist"
Oct 30 20:12:25.536701 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.536617    5958 scope.go:122] "RemoveContainer" containerID="f5f7b43196475efd0db9db8770deabe65b8a5276355d90df18320252ede3dbb2"
Oct 30 20:12:25.536985 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:12:25.536961    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5f7b43196475efd0db9db8770deabe65b8a5276355d90df18320252ede3dbb2\": container with ID starting with f5f7b43196475efd0db9db8770deabe65b8a5276355d90df18320252ede3dbb2 not found: ID does not exist" containerID="f5f7b43196475efd0db9db8770deabe65b8a5276355d90df18320252ede3dbb2"
Oct 30 20:12:25.537048 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.536990    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5f7b43196475efd0db9db8770deabe65b8a5276355d90df18320252ede3dbb2"} err="failed to get container status \"f5f7b43196475efd0db9db8770deabe65b8a5276355d90df18320252ede3dbb2\": rpc error: code = NotFound desc = could not find container \"f5f7b43196475efd0db9db8770deabe65b8a5276355d90df18320252ede3dbb2\": container with ID starting with f5f7b43196475efd0db9db8770deabe65b8a5276355d90df18320252ede3dbb2 not found: ID does not exist"
Oct 30 20:12:25.595833 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.595796    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede036c9-86b3-4d4b-95b0-88570503f1c2-kube-api-access-tz6wk\" (UniqueName: \"kubernetes.io/projected/ede036c9-86b3-4d4b-95b0-88570503f1c2-kube-api-access-tz6wk\") pod \"ede036c9-86b3-4d4b-95b0-88570503f1c2\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") "
Oct 30 20:12:25.596029 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.595885    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ae88c8-fb47-43da-8804-c10a62cb2179-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ae88c8-fb47-43da-8804-c10a62cb2179-maas-api-tls\") pod \"90ae88c8-fb47-43da-8804-c10a62cb2179\" (UID: \"90ae88c8-fb47-43da-8804-c10a62cb2179\") "
Oct 30 20:12:25.596096 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.596022    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-data\") pod \"ede036c9-86b3-4d4b-95b0-88570503f1c2\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") "
Oct 30 20:12:25.596155 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.596109    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-credential-socket\") pod \"ede036c9-86b3-4d4b-95b0-88570503f1c2\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") "
Oct 30 20:12:25.596155 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.596149    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-workload-certs\") pod \"ede036c9-86b3-4d4b-95b0-88570503f1c2\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") "
Oct 30 20:12:25.596363 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.596226    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ae88c8-fb47-43da-8804-c10a62cb2179-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ae88c8-fb47-43da-8804-c10a62cb2179-openshift-service-ca\") pod \"90ae88c8-fb47-43da-8804-c10a62cb2179\" (UID: \"90ae88c8-fb47-43da-8804-c10a62cb2179\") "
Oct 30 20:12:25.596363 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.596288    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-workload-socket\") pod \"ede036c9-86b3-4d4b-95b0-88570503f1c2\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") "
Oct 30 20:12:25.596363 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.596340    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ede036c9-86b3-4d4b-95b0-88570503f1c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede036c9-86b3-4d4b-95b0-88570503f1c2-istiod-ca-cert\") pod \"ede036c9-86b3-4d4b-95b0-88570503f1c2\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") "
Oct 30 20:12:25.596561 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.596367    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-podinfo\") pod \"ede036c9-86b3-4d4b-95b0-88570503f1c2\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") "
Oct 30 20:12:25.596561 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.596497    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-credential-socket" pod "ede036c9-86b3-4d4b-95b0-88570503f1c2" (UID: "ede036c9-86b3-4d4b-95b0-88570503f1c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:25.596561 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.596548    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-envoy\") pod \"ede036c9-86b3-4d4b-95b0-88570503f1c2\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") "
Oct 30 20:12:25.597863 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.597030    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ae88c8-fb47-43da-8804-c10a62cb2179-openshift-service-ca" pod "90ae88c8-fb47-43da-8804-c10a62cb2179" (UID: "90ae88c8-fb47-43da-8804-c10a62cb2179"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:25.597863 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.597189    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ae88c8-fb47-43da-8804-c10a62cb2179-metrics-certs\" (UniqueName: \"kubernetes.io/secret/90ae88c8-fb47-43da-8804-c10a62cb2179-metrics-certs\") pod \"90ae88c8-fb47-43da-8804-c10a62cb2179\" (UID: \"90ae88c8-fb47-43da-8804-c10a62cb2179\") "
Oct 30 20:12:25.597863 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.597284    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-token\" (UniqueName: \"kubernetes.io/projected/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-token\") pod \"ede036c9-86b3-4d4b-95b0-88570503f1c2\" (UID: \"ede036c9-86b3-4d4b-95b0-88570503f1c2\") "
Oct 30 20:12:25.597863 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.597427    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ae88c8-fb47-43da-8804-c10a62cb2179-kube-api-access-9mxfv\" (UniqueName: \"kubernetes.io/projected/90ae88c8-fb47-43da-8804-c10a62cb2179-kube-api-access-9mxfv\") pod \"90ae88c8-fb47-43da-8804-c10a62cb2179\" (UID: \"90ae88c8-fb47-43da-8804-c10a62cb2179\") "
Oct 30 20:12:25.597863 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.597693    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-workload-socket" pod "ede036c9-86b3-4d4b-95b0-88570503f1c2" (UID: "ede036c9-86b3-4d4b-95b0-88570503f1c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:25.598277 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.598264    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede036c9-86b3-4d4b-95b0-88570503f1c2-istiod-ca-cert" pod "ede036c9-86b3-4d4b-95b0-88570503f1c2" (UID: "ede036c9-86b3-4d4b-95b0-88570503f1c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:25.598623 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.598589    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-workload-certs" pod "ede036c9-86b3-4d4b-95b0-88570503f1c2" (UID: "ede036c9-86b3-4d4b-95b0-88570503f1c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:25.598828 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.598791    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ae88c8-fb47-43da-8804-c10a62cb2179-maas-api-tls" pod "90ae88c8-fb47-43da-8804-c10a62cb2179" (UID: "90ae88c8-fb47-43da-8804-c10a62cb2179"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:25.598959 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.598851    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede036c9-86b3-4d4b-95b0-88570503f1c2-kube-api-access-tz6wk" pod "ede036c9-86b3-4d4b-95b0-88570503f1c2" (UID: "ede036c9-86b3-4d4b-95b0-88570503f1c2"). InnerVolumeSpecName "kube-api-access-tz6wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:25.600510 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.599294    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-envoy" pod "ede036c9-86b3-4d4b-95b0-88570503f1c2" (UID: "ede036c9-86b3-4d4b-95b0-88570503f1c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:25.604476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.602572    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-data" pod "ede036c9-86b3-4d4b-95b0-88570503f1c2" (UID: "ede036c9-86b3-4d4b-95b0-88570503f1c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:25.604476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.603628    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-podinfo" pod "ede036c9-86b3-4d4b-95b0-88570503f1c2" (UID: "ede036c9-86b3-4d4b-95b0-88570503f1c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:12:25.604476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.603653    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ae88c8-fb47-43da-8804-c10a62cb2179-kube-api-access-9mxfv" pod "90ae88c8-fb47-43da-8804-c10a62cb2179" (UID: "90ae88c8-fb47-43da-8804-c10a62cb2179"). InnerVolumeSpecName "kube-api-access-9mxfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:25.604476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.603678    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-token" pod "ede036c9-86b3-4d4b-95b0-88570503f1c2" (UID: "ede036c9-86b3-4d4b-95b0-88570503f1c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:25.604476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.603725    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ae88c8-fb47-43da-8804-c10a62cb2179-metrics-certs" pod "90ae88c8-fb47-43da-8804-c10a62cb2179" (UID: "90ae88c8-fb47-43da-8804-c10a62cb2179"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:25.605292 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.605234    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90ae88c8-fb47-43da-8804-c10a62cb2179-metrics-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:25.605373 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.605297    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:25.605373 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.605314    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mxfv\" (UniqueName: \"kubernetes.io/projected/90ae88c8-fb47-43da-8804-c10a62cb2179-kube-api-access-9mxfv\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:25.605373 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.605359    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz6wk\" (UniqueName: \"kubernetes.io/projected/ede036c9-86b3-4d4b-95b0-88570503f1c2-kube-api-access-tz6wk\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:25.605624 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.605373    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ae88c8-fb47-43da-8804-c10a62cb2179-maas-api-tls\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:25.605624 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.605386    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:25.605624 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.605400    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:25.605624 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.605413    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:25.605624 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.605427    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ae88c8-fb47-43da-8804-c10a62cb2179-openshift-service-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:25.605624 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.605443    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:25.605624 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.605517    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede036c9-86b3-4d4b-95b0-88570503f1c2-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:25.605624 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.605529    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:25.605624 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.605541    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede036c9-86b3-4d4b-95b0-88570503f1c2-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:25.953901 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.953848    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"]
Oct 30 20:12:25.957568 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:25.957538    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-80e4d90e-openshift-default-585dc7f4cc-vzc2c"]
Oct 30 20:12:26.203950 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.203907    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f"]
Oct 30 20:12:26.206370 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.204770    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ede036c9-86b3-4d4b-95b0-88570503f1c2" containerName="istio-proxy"
Oct 30 20:12:26.206370 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.204790    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede036c9-86b3-4d4b-95b0-88570503f1c2" containerName="istio-proxy"
Oct 30 20:12:26.206370 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.204808    5958 cpu_manager.go:414] "containerMap: removing container" podUID="7b8f2b7f-2e80-4fde-ac00-ee68816804be" containerName="maas-api"
Oct 30 20:12:26.206370 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.204813    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b8f2b7f-2e80-4fde-ac00-ee68816804be" containerName="maas-api"
Oct 30 20:12:26.206370 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.204827    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ede036c9-86b3-4d4b-95b0-88570503f1c2" containerName="istio-proxy"
Oct 30 20:12:26.206370 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.204833    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede036c9-86b3-4d4b-95b0-88570503f1c2" containerName="istio-proxy"
Oct 30 20:12:26.206370 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.204845    5958 cpu_manager.go:414] "containerMap: removing container" podUID="33bf491a-644a-4a04-b8d5-5ca6842ff3e6" containerName="istio-proxy"
Oct 30 20:12:26.206370 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.204850    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33bf491a-644a-4a04-b8d5-5ca6842ff3e6" containerName="istio-proxy"
Oct 30 20:12:26.206370 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.204857    5958 cpu_manager.go:414] "containerMap: removing container" podUID="90ae88c8-fb47-43da-8804-c10a62cb2179" containerName="maas-api"
Oct 30 20:12:26.206370 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.204861    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ae88c8-fb47-43da-8804-c10a62cb2179" containerName="maas-api"
Oct 30 20:12:26.206370 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.204874    5958 cpu_manager.go:414] "containerMap: removing container" podUID="33bf491a-644a-4a04-b8d5-5ca6842ff3e6" containerName="istio-proxy"
Oct 30 20:12:26.206370 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.204878    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33bf491a-644a-4a04-b8d5-5ca6842ff3e6" containerName="istio-proxy"
Oct 30 20:12:26.206370 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.205005    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede036c9-86b3-4d4b-95b0-88570503f1c2" containerName="istio-proxy"
Oct 30 20:12:26.206370 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.205014    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="33bf491a-644a-4a04-b8d5-5ca6842ff3e6" containerName="istio-proxy"
Oct 30 20:12:26.206370 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.205022    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ae88c8-fb47-43da-8804-c10a62cb2179" containerName="maas-api"
Oct 30 20:12:26.206370 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.205029    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="33bf491a-644a-4a04-b8d5-5ca6842ff3e6" containerName="istio-proxy"
Oct 30 20:12:26.206370 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.205043    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b8f2b7f-2e80-4fde-ac00-ee68816804be" containerName="maas-api"
Oct 30 20:12:26.227325 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.210872    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f"
Oct 30 20:12:26.274475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.274219    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-a2103a\""
Oct 30 20:12:26.274475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.274375    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-a2103a-dockercfg-89kwl\""
Oct 30 20:12:26.285475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.276124    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f"]
Oct 30 20:12:26.365564 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.350491    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f\" (UID: \"b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f"
Oct 30 20:12:26.365564 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.350841    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qck4\" (UniqueName: \"kubernetes.io/projected/b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a-kube-api-access-7qck4\") pod \"payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f\" (UID: \"b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f"
Oct 30 20:12:26.368695 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.368625    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd"]
Oct 30 20:12:26.370341 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.370015    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede036c9-86b3-4d4b-95b0-88570503f1c2" containerName="istio-proxy"
Oct 30 20:12:26.378410 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.376085    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd"
Oct 30 20:12:26.406024 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.400351    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd"]
Oct 30 20:12:26.455893 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.455844    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qck4\" (UniqueName: \"kubernetes.io/projected/b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a-kube-api-access-7qck4\") pod \"payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f\" (UID: \"b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f"
Oct 30 20:12:26.456080 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.456003    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3a634a-354d-44eb-a927-123c5afe109f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd\" (UID: \"3a3a634a-354d-44eb-a927-123c5afe109f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd"
Oct 30 20:12:26.456080 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.456036    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8ptb\" (UniqueName: \"kubernetes.io/projected/3a3a634a-354d-44eb-a927-123c5afe109f-kube-api-access-n8ptb\") pod \"payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd\" (UID: \"3a3a634a-354d-44eb-a927-123c5afe109f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd"
Oct 30 20:12:26.456328 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.456306    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f\" (UID: \"b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f"
Oct 30 20:12:26.457550 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.457521    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f\" (UID: \"b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f"
Oct 30 20:12:26.519217 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.519173    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn" event={"ID":"90ae88c8-fb47-43da-8804-c10a62cb2179","Type":"ContainerDied","Data":"44c74a02584f947540e46c6e54f7a131561d0eec8bfb3cc4e8caf9f8e5392677"}
Oct 30 20:12:26.519729 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.519240    5958 scope.go:122] "RemoveContainer" containerID="c25fff084a384832a3c9796073966dd8b0a83b2931e2117dd89d6b76ff9e1e66"
Oct 30 20:12:26.519729 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.519363    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn"
Oct 30 20:12:26.531240 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.531197    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qck4\" (UniqueName: \"kubernetes.io/projected/b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a-kube-api-access-7qck4\") pod \"payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f\" (UID: \"b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f"
Oct 30 20:12:26.538506 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.538475    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f"
Oct 30 20:12:26.557981 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.557949    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3a634a-354d-44eb-a927-123c5afe109f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd\" (UID: \"3a3a634a-354d-44eb-a927-123c5afe109f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd"
Oct 30 20:12:26.558131 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.557999    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8ptb\" (UniqueName: \"kubernetes.io/projected/3a3a634a-354d-44eb-a927-123c5afe109f-kube-api-access-n8ptb\") pod \"payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd\" (UID: \"3a3a634a-354d-44eb-a927-123c5afe109f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd"
Oct 30 20:12:26.558809 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.558789    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3a634a-354d-44eb-a927-123c5afe109f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd\" (UID: \"3a3a634a-354d-44eb-a927-123c5afe109f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd"
Oct 30 20:12:26.636995 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.636819    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8ptb\" (UniqueName: \"kubernetes.io/projected/3a3a634a-354d-44eb-a927-123c5afe109f-kube-api-access-n8ptb\") pod \"payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd\" (UID: \"3a3a634a-354d-44eb-a927-123c5afe109f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd"
Oct 30 20:12:26.713799 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.713760    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd"
Oct 30 20:12:26.763590 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.761659    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn"]
Oct 30 20:12:26.772706 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:26.772674    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7ca5342-5d4cb48fb7-5ctjn"]
Oct 30 20:12:27.012517 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:27.007597    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f"]
Oct 30 20:12:27.016472 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:12:27.015144    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb60fd1ec_0b05_41d3_a9ba_8d1ed3e5904a.slice/crio-728458105aedebb514126bb5013caa203ca6d307e024b665dd7c26bb8213eb9b WatchSource:0}: Error finding container 728458105aedebb514126bb5013caa203ca6d307e024b665dd7c26bb8213eb9b: Status 404 returned error can't find the container with id 728458105aedebb514126bb5013caa203ca6d307e024b665dd7c26bb8213eb9b
Oct 30 20:12:27.186025 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:12:27.185981    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3a634a_354d_44eb_a927_123c5afe109f.slice/crio-7dfe3e40a195561d3a457917333ff1b11f3d152788d0abb86426bcf9d7e02e2e WatchSource:0}: Error finding container 7dfe3e40a195561d3a457917333ff1b11f3d152788d0abb86426bcf9d7e02e2e: Status 404 returned error can't find the container with id 7dfe3e40a195561d3a457917333ff1b11f3d152788d0abb86426bcf9d7e02e2e
Oct 30 20:12:27.189590 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:27.189558    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd"]
Oct 30 20:12:27.405475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:27.403849    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90ae88c8-fb47-43da-8804-c10a62cb2179" path="/var/lib/kubelet/pods/90ae88c8-fb47-43da-8804-c10a62cb2179/volumes"
Oct 30 20:12:27.405475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:27.404982    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ede036c9-86b3-4d4b-95b0-88570503f1c2" path="/var/lib/kubelet/pods/ede036c9-86b3-4d4b-95b0-88570503f1c2/volumes"
Oct 30 20:12:27.538362 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:27.538270    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f" event={"ID":"b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a","Type":"ContainerStarted","Data":"06097316ee435416be1fcf581464f1177945d442c7bc9034fab7b12a55ff5bc7"}
Oct 30 20:12:27.538362 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:27.538360    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f" event={"ID":"b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a","Type":"ContainerStarted","Data":"728458105aedebb514126bb5013caa203ca6d307e024b665dd7c26bb8213eb9b"}
Oct 30 20:12:27.538812 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:27.538391    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f"
Oct 30 20:12:27.540201 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:27.540168    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd" event={"ID":"3a3a634a-354d-44eb-a927-123c5afe109f","Type":"ContainerStarted","Data":"d93dec3bb905d43029811adc6dfda80b4e11947e0a4e86cbfaab90d7590c0e63"}
Oct 30 20:12:27.540344 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:27.540220    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd" event={"ID":"3a3a634a-354d-44eb-a927-123c5afe109f","Type":"ContainerStarted","Data":"7dfe3e40a195561d3a457917333ff1b11f3d152788d0abb86426bcf9d7e02e2e"}
Oct 30 20:12:27.540344 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:27.540246    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd"
Oct 30 20:12:27.659554 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:27.659504    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f" podStartSLOduration=1.659491861 podStartE2EDuration="1.659491861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:27.655724648 +0000 UTC m=+1786.387940037" watchObservedRunningTime="2026-10-30 20:12:27.659491861 +0000 UTC m=+1786.391707250"
Oct 30 20:12:27.777129 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:27.777071    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd" podStartSLOduration=1.777050185 podStartE2EDuration="1.777050185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:27.775749109 +0000 UTC m=+1786.507964525" watchObservedRunningTime="2026-10-30 20:12:27.777050185 +0000 UTC m=+1786.509265571"
Oct 30 20:12:30.498421 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.498384    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"]
Oct 30 20:12:30.509064 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.509007    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.574606 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.572869    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-37732de7-openshift-default-dockercfg-g5kkn\""
Oct 30 20:12:30.574606 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.574471    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"]
Oct 30 20:12:30.620475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.619032    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.620475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.619131    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.620475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.619183    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aeb303b-220a-4b83-acb6-6c1004a977d1-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.620475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.619274    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.620475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.619327    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.620475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.619404    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.620475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.619444    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.620475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.619991    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nljkw\" (UniqueName: \"kubernetes.io/projected/5aeb303b-220a-4b83-acb6-6c1004a977d1-kube-api-access-nljkw\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.620475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.620052    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.721737 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.721697    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nljkw\" (UniqueName: \"kubernetes.io/projected/5aeb303b-220a-4b83-acb6-6c1004a977d1-kube-api-access-nljkw\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.721924 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.721757    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.721924 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.721803    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.722049 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.721945    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.722049 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.722019    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aeb303b-220a-4b83-acb6-6c1004a977d1-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.722149 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.722123    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.722255 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.722179    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.722322 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.722259    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.722380 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.722338    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.722565 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.722460    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.722565 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.722513    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.722736 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.722598    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.722794 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.722737    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.722794 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.722763    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aeb303b-220a-4b83-acb6-6c1004a977d1-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.724792 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.724767    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.724906 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.724857    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.783349 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.783279    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.787737 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.787709    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nljkw\" (UniqueName: \"kubernetes.io/projected/5aeb303b-220a-4b83-acb6-6c1004a977d1-kube-api-access-nljkw\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:30.889932 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:30.889896    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:31.339935 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:12:31.339889    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aeb303b_220a_4b83_acb6_6c1004a977d1.slice/crio-f1b10a73fba3e18c48e8d5b2da7fe08c7a827f5ecfbe52e62f733cbbbf0d8502 WatchSource:0}: Error finding container f1b10a73fba3e18c48e8d5b2da7fe08c7a827f5ecfbe52e62f733cbbbf0d8502: Status 404 returned error can't find the container with id f1b10a73fba3e18c48e8d5b2da7fe08c7a827f5ecfbe52e62f733cbbbf0d8502
Oct 30 20:12:31.343076 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:31.343037    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:12:31.343205 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:31.343117    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:12:31.343205 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:31.343149    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:12:31.345787 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:31.345744    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"]
Oct 30 20:12:31.618556 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:31.618424    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh" event={"ID":"5aeb303b-220a-4b83-acb6-6c1004a977d1","Type":"ContainerStarted","Data":"addbc3e79ea43fcd344f36a942aa0e9cbb1b3582fb831e936ea79b331d2eece9"}
Oct 30 20:12:31.618961 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:31.618563    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh" event={"ID":"5aeb303b-220a-4b83-acb6-6c1004a977d1","Type":"ContainerStarted","Data":"f1b10a73fba3e18c48e8d5b2da7fe08c7a827f5ecfbe52e62f733cbbbf0d8502"}
Oct 30 20:12:31.750696 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:31.750645    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh" podStartSLOduration=1.750618828 podStartE2EDuration="1.750618828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:31.738951352 +0000 UTC m=+1790.471166740" watchObservedRunningTime="2026-10-30 20:12:31.750618828 +0000 UTC m=+1790.482834219"
Oct 30 20:12:31.878929 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:31.878841    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z"]
Oct 30 20:12:31.884997 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:31.884974    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z"
Oct 30 20:12:31.890725 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:31.890698    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:31.937875 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:31.937844    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-577fb85a-dockercfg-nc9r4\""
Oct 30 20:12:31.940433 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:31.940402    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z"]
Oct 30 20:12:31.941538 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:31.941515    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-577fb85a\""
Oct 30 20:12:31.957548 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:31.957509    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03388cb-31a5-41c1-9997-116ec2be50d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z\" (UID: \"f03388cb-31a5-41c1-9997-116ec2be50d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z"
Oct 30 20:12:31.957873 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:31.957842    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp6c9\" (UniqueName: \"kubernetes.io/projected/f03388cb-31a5-41c1-9997-116ec2be50d9-kube-api-access-zp6c9\") pod \"payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z\" (UID: \"f03388cb-31a5-41c1-9997-116ec2be50d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z"
Oct 30 20:12:32.067776 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:32.065186    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp6c9\" (UniqueName: \"kubernetes.io/projected/f03388cb-31a5-41c1-9997-116ec2be50d9-kube-api-access-zp6c9\") pod \"payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z\" (UID: \"f03388cb-31a5-41c1-9997-116ec2be50d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z"
Oct 30 20:12:32.067776 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:32.065409    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03388cb-31a5-41c1-9997-116ec2be50d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z\" (UID: \"f03388cb-31a5-41c1-9997-116ec2be50d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z"
Oct 30 20:12:32.067776 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:32.066428    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03388cb-31a5-41c1-9997-116ec2be50d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z\" (UID: \"f03388cb-31a5-41c1-9997-116ec2be50d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z"
Oct 30 20:12:32.087767 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:32.087733    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5"]
Oct 30 20:12:32.103156 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:32.103126    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5"]
Oct 30 20:12:32.103397 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:32.103247    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5"
Oct 30 20:12:32.129622 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:32.129042    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp6c9\" (UniqueName: \"kubernetes.io/projected/f03388cb-31a5-41c1-9997-116ec2be50d9-kube-api-access-zp6c9\") pod \"payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z\" (UID: \"f03388cb-31a5-41c1-9997-116ec2be50d9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z"
Oct 30 20:12:32.174985 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:32.174940    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20541f8a-ed94-4723-adc8-f6ee759ca566-plugins-config-volume\") pod \"payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5\" (UID: \"20541f8a-ed94-4723-adc8-f6ee759ca566\") " pod="openshift-ingress/payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5"
Oct 30 20:12:32.175168 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:32.175026    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzw82\" (UniqueName: \"kubernetes.io/projected/20541f8a-ed94-4723-adc8-f6ee759ca566-kube-api-access-bzw82\") pod \"payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5\" (UID: \"20541f8a-ed94-4723-adc8-f6ee759ca566\") " pod="openshift-ingress/payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5"
Oct 30 20:12:32.201669 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:32.201619    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z"
Oct 30 20:12:32.277535 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:32.276468    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20541f8a-ed94-4723-adc8-f6ee759ca566-plugins-config-volume\") pod \"payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5\" (UID: \"20541f8a-ed94-4723-adc8-f6ee759ca566\") " pod="openshift-ingress/payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5"
Oct 30 20:12:32.277535 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:32.276532    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzw82\" (UniqueName: \"kubernetes.io/projected/20541f8a-ed94-4723-adc8-f6ee759ca566-kube-api-access-bzw82\") pod \"payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5\" (UID: \"20541f8a-ed94-4723-adc8-f6ee759ca566\") " pod="openshift-ingress/payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5"
Oct 30 20:12:32.277535 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:32.277502    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20541f8a-ed94-4723-adc8-f6ee759ca566-plugins-config-volume\") pod \"payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5\" (UID: \"20541f8a-ed94-4723-adc8-f6ee759ca566\") " pod="openshift-ingress/payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5"
Oct 30 20:12:32.341421 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:32.341368    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzw82\" (UniqueName: \"kubernetes.io/projected/20541f8a-ed94-4723-adc8-f6ee759ca566-kube-api-access-bzw82\") pod \"payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5\" (UID: \"20541f8a-ed94-4723-adc8-f6ee759ca566\") " pod="openshift-ingress/payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5"
Oct 30 20:12:32.424568 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:32.424485    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5"
Oct 30 20:12:32.606788 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:32.606534    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z"]
Oct 30 20:12:32.608919 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:12:32.608892    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03388cb_31a5_41c1_9997_116ec2be50d9.slice/crio-4693a0ca3d937d5e21dbd57ae6125f85c2e5cb0c8819ff6b4e592367afd80ec9 WatchSource:0}: Error finding container 4693a0ca3d937d5e21dbd57ae6125f85c2e5cb0c8819ff6b4e592367afd80ec9: Status 404 returned error can't find the container with id 4693a0ca3d937d5e21dbd57ae6125f85c2e5cb0c8819ff6b4e592367afd80ec9
Oct 30 20:12:32.628266 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:32.628227    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z" event={"ID":"f03388cb-31a5-41c1-9997-116ec2be50d9","Type":"ContainerStarted","Data":"4693a0ca3d937d5e21dbd57ae6125f85c2e5cb0c8819ff6b4e592367afd80ec9"}
Oct 30 20:12:32.821286 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:32.821254    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5"]
Oct 30 20:12:32.825757 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:12:32.825726    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20541f8a_ed94_4723_adc8_f6ee759ca566.slice/crio-025f31942ad571fb2275cd870dd589c0e766b6546b3406723be1b445acd8949f WatchSource:0}: Error finding container 025f31942ad571fb2275cd870dd589c0e766b6546b3406723be1b445acd8949f: Status 404 returned error can't find the container with id 025f31942ad571fb2275cd870dd589c0e766b6546b3406723be1b445acd8949f
Oct 30 20:12:32.896686 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:32.896663    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:33.639138 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:33.639100    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z" event={"ID":"f03388cb-31a5-41c1-9997-116ec2be50d9","Type":"ContainerStarted","Data":"29c23d1b1ae9f14513cd498a9094827ae996fd4fa9ebf1c648e1ceb78bbde88c"}
Oct 30 20:12:33.639552 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:33.639170    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z"
Oct 30 20:12:33.640811 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:33.640778    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5" event={"ID":"20541f8a-ed94-4723-adc8-f6ee759ca566","Type":"ContainerStarted","Data":"49c75723d7d7da8e471ca208e4dbe0a24db0e5d19265c6ca953a27e04506c239"}
Oct 30 20:12:33.640926 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:33.640833    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5" event={"ID":"20541f8a-ed94-4723-adc8-f6ee759ca566","Type":"ContainerStarted","Data":"025f31942ad571fb2275cd870dd589c0e766b6546b3406723be1b445acd8949f"}
Oct 30 20:12:33.641003 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:33.640983    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:33.641052 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:33.641033    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5"
Oct 30 20:12:33.641909 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:33.641889    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:12:33.756883 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:33.756821    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z" podStartSLOduration=2.756803771 podStartE2EDuration="2.756803771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:33.755026442 +0000 UTC m=+1792.487241832" watchObservedRunningTime="2026-10-30 20:12:33.756803771 +0000 UTC m=+1792.489019162"
Oct 30 20:12:33.875792 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:33.875670    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5" podStartSLOduration=2.875656736 podStartE2EDuration="2.875656736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:33.871208762 +0000 UTC m=+1792.603424155" watchObservedRunningTime="2026-10-30 20:12:33.875656736 +0000 UTC m=+1792.607872125"
Oct 30 20:12:35.094006 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:35.093950    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058eadc3-tp8m7"]
Oct 30 20:12:35.098142 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:35.098117    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058eadc3-tp8m7"]
Oct 30 20:12:35.400850 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:35.400773    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c786d26b-042b-42eb-9520-434658f959d1" path="/var/lib/kubelet/pods/c786d26b-042b-42eb-9520-434658f959d1/volumes"
Oct 30 20:12:36.607809 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:12:36.607760    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:12:38.552102 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:38.552072    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd"
Oct 30 20:12:38.552489 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:38.552148    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f"
Oct 30 20:12:40.005815 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:40.005779    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89"]
Oct 30 20:12:40.012401 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:40.012371    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89"
Oct 30 20:12:40.067828 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:40.067789    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89"]
Oct 30 20:12:40.075740 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:40.075705    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-37732de7-dockercfg-b8hm2\""
Oct 30 20:12:40.076557 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:40.076531    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-37732de7\""
Oct 30 20:12:40.120564 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:40.120523    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bea084f-db30-4c5d-9da2-91143a121dc4-plugins-config-volume\") pod \"payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89\" (UID: \"6bea084f-db30-4c5d-9da2-91143a121dc4\") " pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89"
Oct 30 20:12:40.120754 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:40.120597    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6n4c\" (UniqueName: \"kubernetes.io/projected/6bea084f-db30-4c5d-9da2-91143a121dc4-kube-api-access-w6n4c\") pod \"payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89\" (UID: \"6bea084f-db30-4c5d-9da2-91143a121dc4\") " pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89"
Oct 30 20:12:40.224390 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:40.224354    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bea084f-db30-4c5d-9da2-91143a121dc4-plugins-config-volume\") pod \"payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89\" (UID: \"6bea084f-db30-4c5d-9da2-91143a121dc4\") " pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89"
Oct 30 20:12:40.224780 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:40.224743    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6n4c\" (UniqueName: \"kubernetes.io/projected/6bea084f-db30-4c5d-9da2-91143a121dc4-kube-api-access-w6n4c\") pod \"payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89\" (UID: \"6bea084f-db30-4c5d-9da2-91143a121dc4\") " pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89"
Oct 30 20:12:40.226567 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:40.226281    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bea084f-db30-4c5d-9da2-91143a121dc4-plugins-config-volume\") pod \"payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89\" (UID: \"6bea084f-db30-4c5d-9da2-91143a121dc4\") " pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89"
Oct 30 20:12:40.286920 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:40.286840    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6n4c\" (UniqueName: \"kubernetes.io/projected/6bea084f-db30-4c5d-9da2-91143a121dc4-kube-api-access-w6n4c\") pod \"payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89\" (UID: \"6bea084f-db30-4c5d-9da2-91143a121dc4\") " pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89"
Oct 30 20:12:40.331294 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:40.331261    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89"
Oct 30 20:12:40.724861 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:40.724824    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89"]
Oct 30 20:12:41.716655 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:41.716610    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89" event={"ID":"6bea084f-db30-4c5d-9da2-91143a121dc4","Type":"ContainerStarted","Data":"e4700fbb327ed6615ad65b417ada0c256106dabdf98b36a7fd14bbfcaee8be8a"}
Oct 30 20:12:41.717155 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:41.716857    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89"
Oct 30 20:12:41.717155 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:41.716908    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89" event={"ID":"6bea084f-db30-4c5d-9da2-91143a121dc4","Type":"ContainerStarted","Data":"9c02792d41fc250d52d521154964ca8f6eca406b7c1b5fec434582fa8d199f8a"}
Oct 30 20:12:41.724759 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:41.724736    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 20:12:41.727983 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:41.727964    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 20:12:41.802420 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:41.802377    5958 scope.go:122] "RemoveContainer" containerID="93889a1d428e8443a96204cff1951d122b83da348308fbcf933dcef907882518"
Oct 30 20:12:41.840877 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:41.840814    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89" podStartSLOduration=2.840794208 podStartE2EDuration="2.840794208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:41.836977141 +0000 UTC m=+1800.569192542" watchObservedRunningTime="2026-10-30 20:12:41.840794208 +0000 UTC m=+1800.573009598"
Oct 30 20:12:44.655660 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:44.655626    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z"
Oct 30 20:12:44.656105 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:44.655712    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5"
Oct 30 20:12:52.099613 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:52.099509    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38cbb4d-l9bhc"]
Oct 30 20:12:52.103964 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:52.103930    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38cbb4d-l9bhc"]
Oct 30 20:12:52.727785 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:52.727757    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89"
Oct 30 20:12:53.405399 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.405354    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9f55097-1793-4eeb-9081-ff4c6345c738" path="/var/lib/kubelet/pods/e9f55097-1793-4eeb-9081-ff4c6345c738/volumes"
Oct 30 20:12:53.465263 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.465227    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"]
Oct 30 20:12:53.473919 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.473888    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.539045 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.539009    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"]
Oct 30 20:12:53.573650 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.573609    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtpbf\" (UniqueName: \"kubernetes.io/projected/23b32875-225b-469d-8794-c47053a71ebc-kube-api-access-rtpbf\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.573834 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.573710    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23b32875-225b-469d-8794-c47053a71ebc-istiod-ca-cert\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.573834 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.573769    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-istio-data\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.573947 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.573925    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-istio-envoy\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.574007 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.573992    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-workload-certs\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.574172 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.574146    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23b32875-225b-469d-8794-c47053a71ebc-istio-token\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.574275 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.574259    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-workload-socket\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.574334 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.574302    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-credential-socket\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.574391 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.574360    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23b32875-225b-469d-8794-c47053a71ebc-istio-podinfo\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.577613 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.577594    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-f44ec2-openshift-default-dockercfg-6jvdz\""
Oct 30 20:12:53.676192 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.676105    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-istio-envoy\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.676192 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.676182    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-workload-certs\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.676415 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.676310    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23b32875-225b-469d-8794-c47053a71ebc-istio-token\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.676415 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.676392    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-workload-socket\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.676592 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.676425    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-credential-socket\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.676592 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.676503    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23b32875-225b-469d-8794-c47053a71ebc-istio-podinfo\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.676592 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.676549    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtpbf\" (UniqueName: \"kubernetes.io/projected/23b32875-225b-469d-8794-c47053a71ebc-kube-api-access-rtpbf\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.676743 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.676591    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23b32875-225b-469d-8794-c47053a71ebc-istiod-ca-cert\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.676743 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.676661    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-istio-data\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.676743 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.676673    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-workload-certs\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.676882 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.676834    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-workload-socket\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.677388 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.677362    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-credential-socket\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.677573 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.677552    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-istio-data\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.677648 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.677606    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23b32875-225b-469d-8794-c47053a71ebc-istiod-ca-cert\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.679035 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.678993    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-istio-envoy\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.679825 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.679801    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23b32875-225b-469d-8794-c47053a71ebc-istio-podinfo\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.749184 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.749147    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23b32875-225b-469d-8794-c47053a71ebc-istio-token\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.751579 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.751547    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtpbf\" (UniqueName: \"kubernetes.io/projected/23b32875-225b-469d-8794-c47053a71ebc-kube-api-access-rtpbf\") pod \"e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") " pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:53.794047 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:53.794004    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:54.252260 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:54.252224    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"]
Oct 30 20:12:54.252515 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:12:54.252430    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b32875_225b_469d_8794_c47053a71ebc.slice/crio-5266f30ba77f87f1bfcad198f118c23a5918cce85ee04f733dbab578001c55fd WatchSource:0}: Error finding container 5266f30ba77f87f1bfcad198f118c23a5918cce85ee04f733dbab578001c55fd: Status 404 returned error can't find the container with id 5266f30ba77f87f1bfcad198f118c23a5918cce85ee04f733dbab578001c55fd
Oct 30 20:12:54.255227 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:54.255189    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:12:54.255320 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:54.255274    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:12:54.255398 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:54.255321    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:12:54.840498 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:54.840467    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd" event={"ID":"23b32875-225b-469d-8794-c47053a71ebc","Type":"ContainerStarted","Data":"ab3eb8f380d8b23cc73f3c5f1a7a37c6559614a6200a59a9635ebce44917ebb6"}
Oct 30 20:12:54.840900 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:54.840514    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd" event={"ID":"23b32875-225b-469d-8794-c47053a71ebc","Type":"ContainerStarted","Data":"5266f30ba77f87f1bfcad198f118c23a5918cce85ee04f733dbab578001c55fd"}
Oct 30 20:12:54.966611 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:54.966562    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd" podStartSLOduration=1.9665483799999999 podStartE2EDuration="1.96654838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:54.962526041 +0000 UTC m=+1813.694741456" watchObservedRunningTime="2026-10-30 20:12:54.96654838 +0000 UTC m=+1813.698763769"
Oct 30 20:12:55.794913 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:55.794878    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:55.800319 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:55.800297    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:55.848252 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:55.848225    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:12:55.849054 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:12:55.849033    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:13:00.188501 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:00.188434    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482"]
Oct 30 20:13:00.194435 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:00.194407    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482"
Oct 30 20:13:00.296735 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:00.296694    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f3736ef9\""
Oct 30 20:13:00.296735 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:00.296696    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f3736ef9-dockercfg-smdmw\""
Oct 30 20:13:00.300492 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:00.300440    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482"]
Oct 30 20:13:00.357477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:00.334797    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73694a08-761d-412e-9a30-5c02be8900dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482\" (UID: \"73694a08-761d-412e-9a30-5c02be8900dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482"
Oct 30 20:13:00.357477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:00.335192    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmgr7\" (UniqueName: \"kubernetes.io/projected/73694a08-761d-412e-9a30-5c02be8900dd-kube-api-access-hmgr7\") pod \"payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482\" (UID: \"73694a08-761d-412e-9a30-5c02be8900dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482"
Oct 30 20:13:00.493243 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:00.493092    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73694a08-761d-412e-9a30-5c02be8900dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482\" (UID: \"73694a08-761d-412e-9a30-5c02be8900dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482"
Oct 30 20:13:00.493425 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:00.493298    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmgr7\" (UniqueName: \"kubernetes.io/projected/73694a08-761d-412e-9a30-5c02be8900dd-kube-api-access-hmgr7\") pod \"payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482\" (UID: \"73694a08-761d-412e-9a30-5c02be8900dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482"
Oct 30 20:13:00.494829 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:00.494796    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73694a08-761d-412e-9a30-5c02be8900dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482\" (UID: \"73694a08-761d-412e-9a30-5c02be8900dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482"
Oct 30 20:13:00.555590 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:00.555557    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmgr7\" (UniqueName: \"kubernetes.io/projected/73694a08-761d-412e-9a30-5c02be8900dd-kube-api-access-hmgr7\") pod \"payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482\" (UID: \"73694a08-761d-412e-9a30-5c02be8900dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482"
Oct 30 20:13:00.810803 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:00.810718    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482"
Oct 30 20:13:01.198333 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:01.198299    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482"]
Oct 30 20:13:01.199140 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:13:01.198843    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73694a08_761d_412e_9a30_5c02be8900dd.slice/crio-a130ea4fe56dd758753fc4040d3690ca18f11475bbee7f14750bed67b27235ba WatchSource:0}: Error finding container a130ea4fe56dd758753fc4040d3690ca18f11475bbee7f14750bed67b27235ba: Status 404 returned error can't find the container with id a130ea4fe56dd758753fc4040d3690ca18f11475bbee7f14750bed67b27235ba
Oct 30 20:13:01.951147 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:01.951107    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482" event={"ID":"73694a08-761d-412e-9a30-5c02be8900dd","Type":"ContainerStarted","Data":"0e208d30f988086800eea677204c765204521a97f9c60a5758271d3907d9b88b"}
Oct 30 20:13:01.951376 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:01.951162    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482" event={"ID":"73694a08-761d-412e-9a30-5c02be8900dd","Type":"ContainerStarted","Data":"a130ea4fe56dd758753fc4040d3690ca18f11475bbee7f14750bed67b27235ba"}
Oct 30 20:13:01.951376 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:01.951245    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482"
Oct 30 20:13:02.071743 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:02.071685    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482" podStartSLOduration=2.071671749 podStartE2EDuration="2.071671749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:13:02.067922843 +0000 UTC m=+1820.800138235" watchObservedRunningTime="2026-10-30 20:13:02.071671749 +0000 UTC m=+1820.803887215"
Oct 30 20:13:06.607056 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:13:06.606994    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:13:07.242515 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:07.242476    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6"]
Oct 30 20:13:07.248789 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:07.248755    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6"
Oct 30 20:13:07.297836 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:07.297803    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6\" (UID: \"8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6"
Oct 30 20:13:07.298101 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:07.298078    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgzqq\" (UniqueName: \"kubernetes.io/projected/8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f-kube-api-access-fgzqq\") pod \"payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6\" (UID: \"8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6"
Oct 30 20:13:07.303414 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:07.303385    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-f44ec2\""
Oct 30 20:13:07.303682 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:07.303595    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-f44ec2-dockercfg-87gjr\""
Oct 30 20:13:07.311238 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:07.310755    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6"]
Oct 30 20:13:07.399838 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:07.399806    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6\" (UID: \"8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6"
Oct 30 20:13:07.400059 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:07.400040    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgzqq\" (UniqueName: \"kubernetes.io/projected/8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f-kube-api-access-fgzqq\") pod \"payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6\" (UID: \"8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6"
Oct 30 20:13:07.401420 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:07.401393    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6\" (UID: \"8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6"
Oct 30 20:13:07.463909 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:07.463266    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgzqq\" (UniqueName: \"kubernetes.io/projected/8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f-kube-api-access-fgzqq\") pod \"payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6\" (UID: \"8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6"
Oct 30 20:13:07.564994 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:07.564922    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6"
Oct 30 20:13:07.983018 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:07.982983    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6"]
Oct 30 20:13:07.984114 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:13:07.984090    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b11fbdf_f4e7_4403_b4c5_eb73dc1f034f.slice/crio-62dc0e7eb25e02e888af79fbdc3a7bdc65217ec28e2160e77d54f3b6ae9b04f8 WatchSource:0}: Error finding container 62dc0e7eb25e02e888af79fbdc3a7bdc65217ec28e2160e77d54f3b6ae9b04f8: Status 404 returned error can't find the container with id 62dc0e7eb25e02e888af79fbdc3a7bdc65217ec28e2160e77d54f3b6ae9b04f8
Oct 30 20:13:08.011068 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:08.011039    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6" event={"ID":"8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f","Type":"ContainerStarted","Data":"62dc0e7eb25e02e888af79fbdc3a7bdc65217ec28e2160e77d54f3b6ae9b04f8"}
Oct 30 20:13:08.055401 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:08.055374    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5da4abe-r54zr"]
Oct 30 20:13:08.059589 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:08.059566    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5da4abe-r54zr"]
Oct 30 20:13:09.020864 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:09.020829    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6" event={"ID":"8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f","Type":"ContainerStarted","Data":"56f9a820b840978404faaae17a5816eba7097ea66626c7f298fbcf87fbd1c46f"}
Oct 30 20:13:09.021251 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:09.020906    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6"
Oct 30 20:13:09.140744 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:09.140674    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6" podStartSLOduration=2.140658598 podStartE2EDuration="2.140658598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:13:09.136602274 +0000 UTC m=+1827.868817678" watchObservedRunningTime="2026-10-30 20:13:09.140658598 +0000 UTC m=+1827.872873989"
Oct 30 20:13:09.401678 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:09.401642    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d82aa63-2171-4e98-8861-aedf321fd81f" path="/var/lib/kubelet/pods/2d82aa63-2171-4e98-8861-aedf321fd81f/volumes"
Oct 30 20:13:12.962710 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:12.962679    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482"
Oct 30 20:13:20.032431 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:20.032394    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6"
Oct 30 20:13:21.980605 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:21.980570    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt"]
Oct 30 20:13:21.986530 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:21.986497    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt"
Oct 30 20:13:22.059544 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:22.059509    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzrhm\" (UniqueName: \"kubernetes.io/projected/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-kube-api-access-pzrhm\") pod \"maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt\" (UID: \"e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt"
Oct 30 20:13:22.059544 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:22.059559    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt\" (UID: \"e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt"
Oct 30 20:13:22.059805 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:22.059731    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt\" (UID: \"e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt"
Oct 30 20:13:22.059805 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:22.059785    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt\" (UID: \"e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt"
Oct 30 20:13:22.089400 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:22.089364    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-a2103a-serving-cert\""
Oct 30 20:13:22.095187 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:22.095159    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt"]
Oct 30 20:13:22.162240 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:22.162202    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzrhm\" (UniqueName: \"kubernetes.io/projected/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-kube-api-access-pzrhm\") pod \"maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt\" (UID: \"e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt"
Oct 30 20:13:22.162440 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:22.162257    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt\" (UID: \"e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt"
Oct 30 20:13:22.162440 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:22.162330    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt\" (UID: \"e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt"
Oct 30 20:13:22.162440 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:13:22.162349    5958 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-serving-cert: secret "maas-api-e2e-shared-b-w2-a2103a-serving-cert" not found
Oct 30 20:13:22.162440 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:22.162365    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt\" (UID: \"e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt"
Oct 30 20:13:22.162440 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:13:22.162426    5958 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-maas-api-tls podName:e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb nodeName:}" failed. No retries permitted until 2026-10-30 20:13:22.662403987 +0000 UTC m=+1841.394619404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-maas-api-tls") pod "maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt" (UID: "e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb") : secret "maas-api-e2e-shared-b-w2-a2103a-serving-cert" not found
Oct 30 20:13:22.163084 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:22.163056    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt\" (UID: \"e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt"
Oct 30 20:13:22.165073 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:22.165053    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt\" (UID: \"e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt"
Oct 30 20:13:22.230831 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:22.230748    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzrhm\" (UniqueName: \"kubernetes.io/projected/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-kube-api-access-pzrhm\") pod \"maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt\" (UID: \"e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt"
Oct 30 20:13:22.676876 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:22.676843    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt\" (UID: \"e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt"
Oct 30 20:13:22.679551 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:22.679528    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt\" (UID: \"e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt"
Oct 30 20:13:22.902357 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:22.902317    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt"
Oct 30 20:13:23.302603 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:23.302570    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt"]
Oct 30 20:13:23.302603 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:13:23.302584    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f6b6ff_38d0_4f73_8b80_b9767a2a9ecb.slice/crio-f78875b08d9932c9be49488cc17023d7a34321440d97754195c9b317071d9e20 WatchSource:0}: Error finding container f78875b08d9932c9be49488cc17023d7a34321440d97754195c9b317071d9e20: Status 404 returned error can't find the container with id f78875b08d9932c9be49488cc17023d7a34321440d97754195c9b317071d9e20
Oct 30 20:13:24.156226 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:24.156184    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt" event={"ID":"e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb","Type":"ContainerStarted","Data":"f78875b08d9932c9be49488cc17023d7a34321440d97754195c9b317071d9e20"}
Oct 30 20:13:25.167523 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:25.167481    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt" event={"ID":"e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb","Type":"ContainerStarted","Data":"3c920dded65b63289c86c65cdabdf68a864f32f930c720908c00d652c82077cc"}
Oct 30 20:13:25.167917 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:25.167567    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt"
Oct 30 20:13:25.289390 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:25.289337    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt" podStartSLOduration=3.290866033 podStartE2EDuration="4.289323512s" totalImagesPullingTime="998.457479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:13:25.28630788 +0000 UTC m=+1844.018523269" watchObservedRunningTime="2026-10-30 20:13:25.289323512 +0000 UTC m=+1844.021538900"
Oct 30 20:13:27.117552 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:27.117516    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"]
Oct 30 20:13:27.122620 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:27.122598    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"
Oct 30 20:13:27.176155 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:27.176103    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-577fb85a-serving-cert\""
Oct 30 20:13:27.183129 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:27.183102    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"]
Oct 30 20:13:27.256483 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:27.256423    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64vjn\" (UniqueName: \"kubernetes.io/projected/6a6506dc-92f6-46cd-8e96-030a02603d96-kube-api-access-64vjn\") pod \"maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp\" (UID: \"6a6506dc-92f6-46cd-8e96-030a02603d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"
Oct 30 20:13:27.256642 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:27.256584    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a6506dc-92f6-46cd-8e96-030a02603d96-maas-api-tls\") pod \"maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp\" (UID: \"6a6506dc-92f6-46cd-8e96-030a02603d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"
Oct 30 20:13:27.256857 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:27.256837    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6506dc-92f6-46cd-8e96-030a02603d96-metrics-certs\") pod \"maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp\" (UID: \"6a6506dc-92f6-46cd-8e96-030a02603d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"
Oct 30 20:13:27.257019 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:27.256997    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a6506dc-92f6-46cd-8e96-030a02603d96-openshift-service-ca\") pod \"maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp\" (UID: \"6a6506dc-92f6-46cd-8e96-030a02603d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"
Oct 30 20:13:27.359100 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:27.359058    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a6506dc-92f6-46cd-8e96-030a02603d96-openshift-service-ca\") pod \"maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp\" (UID: \"6a6506dc-92f6-46cd-8e96-030a02603d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"
Oct 30 20:13:27.359296 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:27.359233    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64vjn\" (UniqueName: \"kubernetes.io/projected/6a6506dc-92f6-46cd-8e96-030a02603d96-kube-api-access-64vjn\") pod \"maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp\" (UID: \"6a6506dc-92f6-46cd-8e96-030a02603d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"
Oct 30 20:13:27.359296 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:27.359289    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a6506dc-92f6-46cd-8e96-030a02603d96-maas-api-tls\") pod \"maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp\" (UID: \"6a6506dc-92f6-46cd-8e96-030a02603d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"
Oct 30 20:13:27.360133 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:27.359555    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6506dc-92f6-46cd-8e96-030a02603d96-metrics-certs\") pod \"maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp\" (UID: \"6a6506dc-92f6-46cd-8e96-030a02603d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"
Oct 30 20:13:27.360133 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:13:27.359792    5958 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-serving-cert: secret "maas-api-e2e-ait-577fb85a-serving-cert" not found
Oct 30 20:13:27.360133 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:13:27.359866    5958 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a6506dc-92f6-46cd-8e96-030a02603d96-maas-api-tls podName:6a6506dc-92f6-46cd-8e96-030a02603d96 nodeName:}" failed. No retries permitted until 2026-10-30 20:13:27.859843805 +0000 UTC m=+1846.592059174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a6506dc-92f6-46cd-8e96-030a02603d96-maas-api-tls") pod "maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp" (UID: "6a6506dc-92f6-46cd-8e96-030a02603d96") : secret "maas-api-e2e-ait-577fb85a-serving-cert" not found
Oct 30 20:13:27.360292 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:27.360221    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a6506dc-92f6-46cd-8e96-030a02603d96-openshift-service-ca\") pod \"maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp\" (UID: \"6a6506dc-92f6-46cd-8e96-030a02603d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"
Oct 30 20:13:27.362901 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:27.362876    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6506dc-92f6-46cd-8e96-030a02603d96-metrics-certs\") pod \"maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp\" (UID: \"6a6506dc-92f6-46cd-8e96-030a02603d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"
Oct 30 20:13:27.426731 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:27.426649    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64vjn\" (UniqueName: \"kubernetes.io/projected/6a6506dc-92f6-46cd-8e96-030a02603d96-kube-api-access-64vjn\") pod \"maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp\" (UID: \"6a6506dc-92f6-46cd-8e96-030a02603d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"
Oct 30 20:13:27.867230 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:27.867197    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a6506dc-92f6-46cd-8e96-030a02603d96-maas-api-tls\") pod \"maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp\" (UID: \"6a6506dc-92f6-46cd-8e96-030a02603d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"
Oct 30 20:13:27.867378 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:13:27.867345    5958 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-serving-cert: secret "maas-api-e2e-ait-577fb85a-serving-cert" not found
Oct 30 20:13:27.867436 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:13:27.867407    5958 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a6506dc-92f6-46cd-8e96-030a02603d96-maas-api-tls podName:6a6506dc-92f6-46cd-8e96-030a02603d96 nodeName:}" failed. No retries permitted until 2026-10-30 20:13:28.867391241 +0000 UTC m=+1847.599606609 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a6506dc-92f6-46cd-8e96-030a02603d96-maas-api-tls") pod "maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp" (UID: "6a6506dc-92f6-46cd-8e96-030a02603d96") : secret "maas-api-e2e-ait-577fb85a-serving-cert" not found
Oct 30 20:13:28.886786 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:28.881593    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a6506dc-92f6-46cd-8e96-030a02603d96-maas-api-tls\") pod \"maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp\" (UID: \"6a6506dc-92f6-46cd-8e96-030a02603d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"
Oct 30 20:13:28.887252 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:28.887135    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a6506dc-92f6-46cd-8e96-030a02603d96-maas-api-tls\") pod \"maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp\" (UID: \"6a6506dc-92f6-46cd-8e96-030a02603d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"
Oct 30 20:13:28.936580 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:28.936551    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"
Oct 30 20:13:29.323851 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:29.323817    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"]
Oct 30 20:13:29.326937 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:13:29.326904    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6506dc_92f6_46cd_8e96_030a02603d96.slice/crio-6dafe36687fc50de21bf08009ff0867861aa88cb63bc22ed802756f18ff4570c WatchSource:0}: Error finding container 6dafe36687fc50de21bf08009ff0867861aa88cb63bc22ed802756f18ff4570c: Status 404 returned error can't find the container with id 6dafe36687fc50de21bf08009ff0867861aa88cb63bc22ed802756f18ff4570c
Oct 30 20:13:30.214354 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:30.214327    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp" event={"ID":"6a6506dc-92f6-46cd-8e96-030a02603d96","Type":"ContainerStarted","Data":"6dafe36687fc50de21bf08009ff0867861aa88cb63bc22ed802756f18ff4570c"}
Oct 30 20:13:31.181412 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:31.181380    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt"
Oct 30 20:13:31.224039 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:31.224003    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp" event={"ID":"6a6506dc-92f6-46cd-8e96-030a02603d96","Type":"ContainerStarted","Data":"0382469cf03112471d31ab82ec28375f63e1040de168acf57edd8e855346aec9"}
Oct 30 20:13:31.224394 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:31.224098    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"
Oct 30 20:13:31.421222 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:31.421161    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp" podStartSLOduration=3.613568883 podStartE2EDuration="4.421143726s" totalImagesPullingTime="807.574843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:13:31.417941322 +0000 UTC m=+1850.150156724" watchObservedRunningTime="2026-10-30 20:13:31.421143726 +0000 UTC m=+1850.153359116"
Oct 30 20:13:36.608152 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:13:36.608102    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:13:37.237976 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:37.237948    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"
Oct 30 20:13:41.898256 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:41.898220    5958 scope.go:122] "RemoveContainer" containerID="25d540e068026222690a8cb912c0d86c2e31ca83ea1340c12060c89e1f78348b"
Oct 30 20:13:41.909217 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:41.909189    5958 scope.go:122] "RemoveContainer" containerID="a12a5d7582cd0b396392178d0f264c7e97b5149b2b7007ad6d77275bab3d6ea2"
Oct 30 20:13:48.067508 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:48.067472    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8096ef7-k4w6w"]
Oct 30 20:13:48.071865 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:48.071838    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8096ef7-k4w6w"]
Oct 30 20:13:49.405475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:49.405410    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be91550e-f729-4788-8e76-70aea3d869c6" path="/var/lib/kubelet/pods/be91550e-f729-4788-8e76-70aea3d869c6/volumes"
Oct 30 20:13:58.115825 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:58.115789    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46952085-f6fjz"]
Oct 30 20:13:58.121126 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:58.121104    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46952085-f6fjz"]
Oct 30 20:13:59.059203 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:59.059165    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b1cf7b2-hhgt2"]
Oct 30 20:13:59.111232 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:59.111194    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b1cf7b2-hhgt2"]
Oct 30 20:13:59.402444 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:59.402412    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00217548-5622-4e86-b9f0-3ea049703cd8" path="/var/lib/kubelet/pods/00217548-5622-4e86-b9f0-3ea049703cd8/volumes"
Oct 30 20:13:59.403037 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:13:59.403017    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efb6f6f8-5072-4cb4-bb24-73ce5f4f1509" path="/var/lib/kubelet/pods/efb6f6f8-5072-4cb4-bb24-73ce5f4f1509/volumes"
Oct 30 20:14:06.609834 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:14:06.609793    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:14:12.103084 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:12.103044    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5be6971c-dgxhl"]
Oct 30 20:14:12.107244 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:12.107220    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5be6971c-dgxhl"]
Oct 30 20:14:13.401707 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:13.401676    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd" path="/var/lib/kubelet/pods/1eacc5c0-ca6a-4a8e-b336-a1d48627e3dd/volumes"
Oct 30 20:14:36.607143 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:14:36.607104    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:14:41.945508 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:41.945469    5958 scope.go:122] "RemoveContainer" containerID="74f5ec1e219c06a3ae044066e7185de093f3414f7886d01a6d3019b1101a6d07"
Oct 30 20:14:41.958183 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:41.958157    5958 scope.go:122] "RemoveContainer" containerID="9eac3ce5689a0b326b25f488d0750a731b3de4dac3884cf0da26f090de7caf73"
Oct 30 20:14:44.913399 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:44.913370    5958 generic.go:356] "Generic (PLEG): container finished" podID="5aeb303b-220a-4b83-acb6-6c1004a977d1" containerID="addbc3e79ea43fcd344f36a942aa0e9cbb1b3582fb831e936ea79b331d2eece9" exitCode=0
Oct 30 20:14:44.913796 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:44.913441    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh" event={"ID":"5aeb303b-220a-4b83-acb6-6c1004a977d1","Type":"ContainerDied","Data":"addbc3e79ea43fcd344f36a942aa0e9cbb1b3582fb831e936ea79b331d2eece9"}
Oct 30 20:14:44.913928 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:44.913909    5958 scope.go:122] "RemoveContainer" containerID="addbc3e79ea43fcd344f36a942aa0e9cbb1b3582fb831e936ea79b331d2eece9"
Oct 30 20:14:44.915560 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:44.915528    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:14:44.915648 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:44.915608    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:14:44.915648 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:44.915644    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:14:45.923510 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:45.923438    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh" event={"ID":"5aeb303b-220a-4b83-acb6-6c1004a977d1","Type":"ContainerStarted","Data":"c7c9950a63166d7b3f63c8eda06afef9617a8361f10e0975bef3bba17089127f"}
Oct 30 20:14:46.890508 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:46.890477    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:14:46.891366 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:46.891337    5958 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 30 20:14:46.891499 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:46.891407    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh" podUID="5aeb303b-220a-4b83-acb6-6c1004a977d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 30 20:14:47.890473 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:47.890420    5958 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 30 20:14:47.890903 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:47.890526    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh" podUID="5aeb303b-220a-4b83-acb6-6c1004a977d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 30 20:14:48.648483 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:48.647772    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:14:48.890758 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:48.890727    5958 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 30 20:14:48.891133 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:48.890777    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh" podUID="5aeb303b-220a-4b83-acb6-6c1004a977d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 30 20:14:49.890912 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:49.890878    5958 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 30 20:14:49.891303 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:49.890941    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh" podUID="5aeb303b-220a-4b83-acb6-6c1004a977d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 30 20:14:50.891159 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:50.891127    5958 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 30 20:14:50.891582 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:50.891181    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh" podUID="5aeb303b-220a-4b83-acb6-6c1004a977d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 30 20:14:51.895098 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:51.895067    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:14:51.896205 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:14:51.896182    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:15:00.211677 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.211644    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75"]
Oct 30 20:15:00.216310 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.216288    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75"
Oct 30 20:15:00.255292 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.255262    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc"]
Oct 30 20:15:00.258796 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.258770    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc"
Oct 30 20:15:00.274978 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.274955    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:15:00.277115 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.277069    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7"]
Oct 30 20:15:00.281129 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.281111    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7"
Oct 30 20:15:00.307242 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.307208    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75"]
Oct 30 20:15:00.322111 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.322088    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c"]
Oct 30 20:15:00.349412 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.349379    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl"]
Oct 30 20:15:00.350359 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.350337    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c"
Oct 30 20:15:00.367791 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.367766    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9"]
Oct 30 20:15:00.368161 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.368138    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl"
Oct 30 20:15:00.379631 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.379594    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc"]
Oct 30 20:15:00.379804 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.379642    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7"]
Oct 30 20:15:00.379804 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.379668    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c"]
Oct 30 20:15:00.379804 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.379754    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9"
Oct 30 20:15:00.385085 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.385053    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl"]
Oct 30 20:15:00.393783 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.393759    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2b023d-3aaf-47f7-8fa2-385d0ab430a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7\" (UID: \"ea2b023d-3aaf-47f7-8fa2-385d0ab430a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7"
Oct 30 20:15:00.393891 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.393842    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpss9\" (UniqueName: \"kubernetes.io/projected/a820cb88-eb76-459c-bb08-28837aa714a1-kube-api-access-mpss9\") pod \"maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75\" (UID: \"a820cb88-eb76-459c-bb08-28837aa714a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75"
Oct 30 20:15:00.394033 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.394012    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdxbn\" (UniqueName: \"kubernetes.io/projected/b7fae60d-346d-4d4a-8851-82586d2ddf65-kube-api-access-zdxbn\") pod \"maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc\" (UID: \"b7fae60d-346d-4d4a-8851-82586d2ddf65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc"
Oct 30 20:15:00.394104 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.394075    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7fae60d-346d-4d4a-8851-82586d2ddf65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc\" (UID: \"b7fae60d-346d-4d4a-8851-82586d2ddf65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc"
Oct 30 20:15:00.394206 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.394182    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs78q\" (UniqueName: \"kubernetes.io/projected/ea2b023d-3aaf-47f7-8fa2-385d0ab430a5-kube-api-access-cs78q\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7\" (UID: \"ea2b023d-3aaf-47f7-8fa2-385d0ab430a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7"
Oct 30 20:15:00.394268 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.394254    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a820cb88-eb76-459c-bb08-28837aa714a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75\" (UID: \"a820cb88-eb76-459c-bb08-28837aa714a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75"
Oct 30 20:15:00.428153 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.428120    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9"]
Oct 30 20:15:00.495664 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.495603    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs78q\" (UniqueName: \"kubernetes.io/projected/ea2b023d-3aaf-47f7-8fa2-385d0ab430a5-kube-api-access-cs78q\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7\" (UID: \"ea2b023d-3aaf-47f7-8fa2-385d0ab430a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7"
Oct 30 20:15:00.495664 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.495640    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c85z\" (UniqueName: \"kubernetes.io/projected/e20ab886-1287-453f-baea-c74e65e5ccad-kube-api-access-8c85z\") pod \"maas-api-key-cleanup-29889855-wr98c\" (UID: \"e20ab886-1287-453f-baea-c74e65e5ccad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c"
Oct 30 20:15:00.495839 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.495676    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a820cb88-eb76-459c-bb08-28837aa714a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75\" (UID: \"a820cb88-eb76-459c-bb08-28837aa714a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75"
Oct 30 20:15:00.495839 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.495728    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e20ab886-1287-453f-baea-c74e65e5ccad-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-wr98c\" (UID: \"e20ab886-1287-453f-baea-c74e65e5ccad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c"
Oct 30 20:15:00.495958 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.495889    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed65872e-55ca-416c-bf1d-05e1bc0047b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9\" (UID: \"ed65872e-55ca-416c-bf1d-05e1bc0047b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9"
Oct 30 20:15:00.496017 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.495996    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2b023d-3aaf-47f7-8fa2-385d0ab430a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7\" (UID: \"ea2b023d-3aaf-47f7-8fa2-385d0ab430a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7"
Oct 30 20:15:00.496117 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.496102    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpss9\" (UniqueName: \"kubernetes.io/projected/a820cb88-eb76-459c-bb08-28837aa714a1-kube-api-access-mpss9\") pod \"maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75\" (UID: \"a820cb88-eb76-459c-bb08-28837aa714a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75"
Oct 30 20:15:00.496189 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.496173    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw6qn\" (UniqueName: \"kubernetes.io/projected/ed65872e-55ca-416c-bf1d-05e1bc0047b5-kube-api-access-cw6qn\") pod \"maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9\" (UID: \"ed65872e-55ca-416c-bf1d-05e1bc0047b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9"
Oct 30 20:15:00.496313 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.496293    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2103c1-53ee-4aaf-ab8e-75c5c93390a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl\" (UID: \"5c2103c1-53ee-4aaf-ab8e-75c5c93390a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl"
Oct 30 20:15:00.496313 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.496306    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a820cb88-eb76-459c-bb08-28837aa714a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75\" (UID: \"a820cb88-eb76-459c-bb08-28837aa714a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75"
Oct 30 20:15:00.496426 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.496323    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25jml\" (UniqueName: \"kubernetes.io/projected/5c2103c1-53ee-4aaf-ab8e-75c5c93390a1-kube-api-access-25jml\") pod \"maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl\" (UID: \"5c2103c1-53ee-4aaf-ab8e-75c5c93390a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl"
Oct 30 20:15:00.496426 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.496404    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdxbn\" (UniqueName: \"kubernetes.io/projected/b7fae60d-346d-4d4a-8851-82586d2ddf65-kube-api-access-zdxbn\") pod \"maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc\" (UID: \"b7fae60d-346d-4d4a-8851-82586d2ddf65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc"
Oct 30 20:15:00.496522 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.496495    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7fae60d-346d-4d4a-8851-82586d2ddf65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc\" (UID: \"b7fae60d-346d-4d4a-8851-82586d2ddf65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc"
Oct 30 20:15:00.496627 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.496606    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2b023d-3aaf-47f7-8fa2-385d0ab430a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7\" (UID: \"ea2b023d-3aaf-47f7-8fa2-385d0ab430a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7"
Oct 30 20:15:00.497029 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.497003    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7fae60d-346d-4d4a-8851-82586d2ddf65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc\" (UID: \"b7fae60d-346d-4d4a-8851-82586d2ddf65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc"
Oct 30 20:15:00.557085 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.557064    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpss9\" (UniqueName: \"kubernetes.io/projected/a820cb88-eb76-459c-bb08-28837aa714a1-kube-api-access-mpss9\") pod \"maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75\" (UID: \"a820cb88-eb76-459c-bb08-28837aa714a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75"
Oct 30 20:15:00.557314 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.557286    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdxbn\" (UniqueName: \"kubernetes.io/projected/b7fae60d-346d-4d4a-8851-82586d2ddf65-kube-api-access-zdxbn\") pod \"maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc\" (UID: \"b7fae60d-346d-4d4a-8851-82586d2ddf65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc"
Oct 30 20:15:00.557684 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.557667    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs78q\" (UniqueName: \"kubernetes.io/projected/ea2b023d-3aaf-47f7-8fa2-385d0ab430a5-kube-api-access-cs78q\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7\" (UID: \"ea2b023d-3aaf-47f7-8fa2-385d0ab430a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7"
Oct 30 20:15:00.579251 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.579231    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc"
Oct 30 20:15:00.596801 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.596776    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7"
Oct 30 20:15:00.597304 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.597285    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e20ab886-1287-453f-baea-c74e65e5ccad-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-wr98c\" (UID: \"e20ab886-1287-453f-baea-c74e65e5ccad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c"
Oct 30 20:15:00.597355 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.597334    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed65872e-55ca-416c-bf1d-05e1bc0047b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9\" (UID: \"ed65872e-55ca-416c-bf1d-05e1bc0047b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9"
Oct 30 20:15:00.597493 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.597477    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw6qn\" (UniqueName: \"kubernetes.io/projected/ed65872e-55ca-416c-bf1d-05e1bc0047b5-kube-api-access-cw6qn\") pod \"maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9\" (UID: \"ed65872e-55ca-416c-bf1d-05e1bc0047b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9"
Oct 30 20:15:00.597562 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.597546    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2103c1-53ee-4aaf-ab8e-75c5c93390a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl\" (UID: \"5c2103c1-53ee-4aaf-ab8e-75c5c93390a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl"
Oct 30 20:15:00.597605 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.597576    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25jml\" (UniqueName: \"kubernetes.io/projected/5c2103c1-53ee-4aaf-ab8e-75c5c93390a1-kube-api-access-25jml\") pod \"maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl\" (UID: \"5c2103c1-53ee-4aaf-ab8e-75c5c93390a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl"
Oct 30 20:15:00.597702 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.597683    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c85z\" (UniqueName: \"kubernetes.io/projected/e20ab886-1287-453f-baea-c74e65e5ccad-kube-api-access-8c85z\") pod \"maas-api-key-cleanup-29889855-wr98c\" (UID: \"e20ab886-1287-453f-baea-c74e65e5ccad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c"
Oct 30 20:15:00.598119 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.598093    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e20ab886-1287-453f-baea-c74e65e5ccad-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-wr98c\" (UID: \"e20ab886-1287-453f-baea-c74e65e5ccad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c"
Oct 30 20:15:00.598219 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.598134    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed65872e-55ca-416c-bf1d-05e1bc0047b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9\" (UID: \"ed65872e-55ca-416c-bf1d-05e1bc0047b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9"
Oct 30 20:15:00.598311 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.598280    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2103c1-53ee-4aaf-ab8e-75c5c93390a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl\" (UID: \"5c2103c1-53ee-4aaf-ab8e-75c5c93390a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl"
Oct 30 20:15:00.679977 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.679912    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c85z\" (UniqueName: \"kubernetes.io/projected/e20ab886-1287-453f-baea-c74e65e5ccad-kube-api-access-8c85z\") pod \"maas-api-key-cleanup-29889855-wr98c\" (UID: \"e20ab886-1287-453f-baea-c74e65e5ccad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c"
Oct 30 20:15:00.680844 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.680789    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25jml\" (UniqueName: \"kubernetes.io/projected/5c2103c1-53ee-4aaf-ab8e-75c5c93390a1-kube-api-access-25jml\") pod \"maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl\" (UID: \"5c2103c1-53ee-4aaf-ab8e-75c5c93390a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl"
Oct 30 20:15:00.689846 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.689709    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw6qn\" (UniqueName: \"kubernetes.io/projected/ed65872e-55ca-416c-bf1d-05e1bc0047b5-kube-api-access-cw6qn\") pod \"maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9\" (UID: \"ed65872e-55ca-416c-bf1d-05e1bc0047b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9"
Oct 30 20:15:00.690196 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.689999    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c"
Oct 30 20:15:00.696662 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.696634    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl"
Oct 30 20:15:00.705823 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.705405    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9"
Oct 30 20:15:00.832631 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:00.832185    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75"
Oct 30 20:15:01.019306 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:15:01.019263    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7fae60d_346d_4d4a_8851_82586d2ddf65.slice/crio-07e979db704832a57e4c316c54e45f0563754ad0cac8e3e46b9e8bf704722bcb WatchSource:0}: Error finding container 07e979db704832a57e4c316c54e45f0563754ad0cac8e3e46b9e8bf704722bcb: Status 404 returned error can't find the container with id 07e979db704832a57e4c316c54e45f0563754ad0cac8e3e46b9e8bf704722bcb
Oct 30 20:15:01.021349 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:01.021320    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc"]
Oct 30 20:15:01.063196 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:01.063173    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7"]
Oct 30 20:15:01.084081 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:01.084054    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7" event={"ID":"ea2b023d-3aaf-47f7-8fa2-385d0ab430a5","Type":"ContainerStarted","Data":"6e35a24f3f7f4e5a38f655f94dfb017d99e7cf9338e0b06615a4b170ddc53eb6"}
Oct 30 20:15:01.085630 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:01.085607    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc" event={"ID":"b7fae60d-346d-4d4a-8851-82586d2ddf65","Type":"ContainerStarted","Data":"07e979db704832a57e4c316c54e45f0563754ad0cac8e3e46b9e8bf704722bcb"}
Oct 30 20:15:01.130810 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:01.130787    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl"]
Oct 30 20:15:01.130897 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:15:01.130788    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2103c1_53ee_4aaf_ab8e_75c5c93390a1.slice/crio-f8e7567046060abfecaecfbea8916e37f08bb6d64e1359f1b22c316164eb76e9 WatchSource:0}: Error finding container f8e7567046060abfecaecfbea8916e37f08bb6d64e1359f1b22c316164eb76e9: Status 404 returned error can't find the container with id f8e7567046060abfecaecfbea8916e37f08bb6d64e1359f1b22c316164eb76e9
Oct 30 20:15:01.175496 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:01.175470    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c"]
Oct 30 20:15:01.181096 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:01.181073    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9"]
Oct 30 20:15:01.240106 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:15:01.240081    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode20ab886_1287_453f_baea_c74e65e5ccad.slice/crio-3c15ec93803d27ba68c3f93947ec872fff589531a3ee1f9a24d2813208204aa3 WatchSource:0}: Error finding container 3c15ec93803d27ba68c3f93947ec872fff589531a3ee1f9a24d2813208204aa3: Status 404 returned error can't find the container with id 3c15ec93803d27ba68c3f93947ec872fff589531a3ee1f9a24d2813208204aa3
Oct 30 20:15:01.240992 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:15:01.240966    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded65872e_55ca_416c_bf1d_05e1bc0047b5.slice/crio-a3efdfbe699985b660a10ecd5c32e0a807081d3d5c411e540b0a2b3cda0b1e03 WatchSource:0}: Error finding container a3efdfbe699985b660a10ecd5c32e0a807081d3d5c411e540b0a2b3cda0b1e03: Status 404 returned error can't find the container with id a3efdfbe699985b660a10ecd5c32e0a807081d3d5c411e540b0a2b3cda0b1e03
Oct 30 20:15:01.247657 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:15:01.247629    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda820cb88_eb76_459c_bb08_28837aa714a1.slice/crio-3efec6e2ad0375b48558de473c917abbb8cd6441602b2dec29ce164be354e813 WatchSource:0}: Error finding container 3efec6e2ad0375b48558de473c917abbb8cd6441602b2dec29ce164be354e813: Status 404 returned error can't find the container with id 3efec6e2ad0375b48558de473c917abbb8cd6441602b2dec29ce164be354e813
Oct 30 20:15:01.248007 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:01.247989    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75"]
Oct 30 20:15:02.096669 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:02.096634    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7" event={"ID":"ea2b023d-3aaf-47f7-8fa2-385d0ab430a5","Type":"ContainerStarted","Data":"5c6c797779faa182aff840bd5454041048ae56878bc3aba2b96b3b04c63624f4"}
Oct 30 20:15:02.098361 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:02.098335    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc" event={"ID":"b7fae60d-346d-4d4a-8851-82586d2ddf65","Type":"ContainerStarted","Data":"e13ebbc84c604989cf4f763cd4e7fe92d6988da0c57e2a957ad5b3b8cff41b4a"}
Oct 30 20:15:02.100009 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:02.099978    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75" event={"ID":"a820cb88-eb76-459c-bb08-28837aa714a1","Type":"ContainerStarted","Data":"26e605795fdfd1c66aa2dbb35d2267a9fa777e6569065b2477f6e68ad7bc60c6"}
Oct 30 20:15:02.100113 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:02.100023    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75" event={"ID":"a820cb88-eb76-459c-bb08-28837aa714a1","Type":"ContainerStarted","Data":"3efec6e2ad0375b48558de473c917abbb8cd6441602b2dec29ce164be354e813"}
Oct 30 20:15:02.101600 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:02.101577    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl" event={"ID":"5c2103c1-53ee-4aaf-ab8e-75c5c93390a1","Type":"ContainerStarted","Data":"dddc898421a27b3c78cf2c024582a2af32647a48d69d3b6c9d25c63c4faed7c1"}
Oct 30 20:15:02.101703 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:02.101619    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl" event={"ID":"5c2103c1-53ee-4aaf-ab8e-75c5c93390a1","Type":"ContainerStarted","Data":"f8e7567046060abfecaecfbea8916e37f08bb6d64e1359f1b22c316164eb76e9"}
Oct 30 20:15:02.103263 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:02.103242    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c" event={"ID":"e20ab886-1287-453f-baea-c74e65e5ccad","Type":"ContainerStarted","Data":"04b11d974e2e95756fb30c944d9e1c27040e98a1a8b63efdee4abccec6d8fed4"}
Oct 30 20:15:02.103347 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:02.103284    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c" event={"ID":"e20ab886-1287-453f-baea-c74e65e5ccad","Type":"ContainerStarted","Data":"3c15ec93803d27ba68c3f93947ec872fff589531a3ee1f9a24d2813208204aa3"}
Oct 30 20:15:02.104838 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:02.104817    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9" event={"ID":"ed65872e-55ca-416c-bf1d-05e1bc0047b5","Type":"ContainerStarted","Data":"57b96604017eb994e0872642b31b4c75c2d15c466ed9ff9aec1ef397fb26862a"}
Oct 30 20:15:02.104948 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:02.104855    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9" event={"ID":"ed65872e-55ca-416c-bf1d-05e1bc0047b5","Type":"ContainerStarted","Data":"a3efdfbe699985b660a10ecd5c32e0a807081d3d5c411e540b0a2b3cda0b1e03"}
Oct 30 20:15:02.214931 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:02.214837    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7" podStartSLOduration=2.214825534 podStartE2EDuration="2.214825534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.212175693 +0000 UTC m=+1940.944391101" watchObservedRunningTime="2026-10-30 20:15:02.214825534 +0000 UTC m=+1940.947040959"
Oct 30 20:15:02.333275 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:02.333215    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9" podStartSLOduration=2.333200285 podStartE2EDuration="2.333200285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.330232064 +0000 UTC m=+1941.062447454" watchObservedRunningTime="2026-10-30 20:15:02.333200285 +0000 UTC m=+1941.065415674"
Oct 30 20:15:02.483297 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:02.455226    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc" podStartSLOduration=2.455206278 podStartE2EDuration="2.455206278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.447910813 +0000 UTC m=+1941.180126206" watchObservedRunningTime="2026-10-30 20:15:02.455206278 +0000 UTC m=+1941.187421668"
Oct 30 20:15:02.567797 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:02.567741    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c" podStartSLOduration=2.567724277 podStartE2EDuration="2.567724277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.566566568 +0000 UTC m=+1941.298781957" watchObservedRunningTime="2026-10-30 20:15:02.567724277 +0000 UTC m=+1941.299939667"
Oct 30 20:15:02.691127 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:02.691066    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl" podStartSLOduration=2.691051246 podStartE2EDuration="2.691051246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.681351853 +0000 UTC m=+1941.413567247" watchObservedRunningTime="2026-10-30 20:15:02.691051246 +0000 UTC m=+1941.423266635"
Oct 30 20:15:02.830546 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:02.829967    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75" podStartSLOduration=2.829947544 podStartE2EDuration="2.829947544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.821670764 +0000 UTC m=+1941.553886181" watchObservedRunningTime="2026-10-30 20:15:02.829947544 +0000 UTC m=+1941.562162934"
Oct 30 20:15:03.122283 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:03.122191    5958 generic.go:356] "Generic (PLEG): container finished" podID="e20ab886-1287-453f-baea-c74e65e5ccad" containerID="04b11d974e2e95756fb30c944d9e1c27040e98a1a8b63efdee4abccec6d8fed4" exitCode=60
Oct 30 20:15:03.122611 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:03.122235    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c" event={"ID":"e20ab886-1287-453f-baea-c74e65e5ccad","Type":"ContainerDied","Data":"04b11d974e2e95756fb30c944d9e1c27040e98a1a8b63efdee4abccec6d8fed4"}
Oct 30 20:15:03.122768 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:03.122728    5958 scope.go:122] "RemoveContainer" containerID="04b11d974e2e95756fb30c944d9e1c27040e98a1a8b63efdee4abccec6d8fed4"
Oct 30 20:15:03.125237 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:03.125215    5958 generic.go:356] "Generic (PLEG): container finished" podID="ed65872e-55ca-416c-bf1d-05e1bc0047b5" containerID="57b96604017eb994e0872642b31b4c75c2d15c466ed9ff9aec1ef397fb26862a" exitCode=60
Oct 30 20:15:03.125348 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:03.125309    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9" event={"ID":"ed65872e-55ca-416c-bf1d-05e1bc0047b5","Type":"ContainerDied","Data":"57b96604017eb994e0872642b31b4c75c2d15c466ed9ff9aec1ef397fb26862a"}
Oct 30 20:15:03.125698 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:03.125676    5958 scope.go:122] "RemoveContainer" containerID="57b96604017eb994e0872642b31b4c75c2d15c466ed9ff9aec1ef397fb26862a"
Oct 30 20:15:03.130850 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:03.130679    5958 generic.go:356] "Generic (PLEG): container finished" podID="ea2b023d-3aaf-47f7-8fa2-385d0ab430a5" containerID="5c6c797779faa182aff840bd5454041048ae56878bc3aba2b96b3b04c63624f4" exitCode=60
Oct 30 20:15:03.130850 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:03.130780    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7" event={"ID":"ea2b023d-3aaf-47f7-8fa2-385d0ab430a5","Type":"ContainerDied","Data":"5c6c797779faa182aff840bd5454041048ae56878bc3aba2b96b3b04c63624f4"}
Oct 30 20:15:03.131057 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:03.131033    5958 scope.go:122] "RemoveContainer" containerID="5c6c797779faa182aff840bd5454041048ae56878bc3aba2b96b3b04c63624f4"
Oct 30 20:15:03.133343 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:03.133319    5958 generic.go:356] "Generic (PLEG): container finished" podID="b7fae60d-346d-4d4a-8851-82586d2ddf65" containerID="e13ebbc84c604989cf4f763cd4e7fe92d6988da0c57e2a957ad5b3b8cff41b4a" exitCode=60
Oct 30 20:15:03.133437 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:03.133343    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc" event={"ID":"b7fae60d-346d-4d4a-8851-82586d2ddf65","Type":"ContainerDied","Data":"e13ebbc84c604989cf4f763cd4e7fe92d6988da0c57e2a957ad5b3b8cff41b4a"}
Oct 30 20:15:03.133831 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:03.133803    5958 scope.go:122] "RemoveContainer" containerID="e13ebbc84c604989cf4f763cd4e7fe92d6988da0c57e2a957ad5b3b8cff41b4a"
Oct 30 20:15:03.135954 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:03.135927    5958 generic.go:356] "Generic (PLEG): container finished" podID="a820cb88-eb76-459c-bb08-28837aa714a1" containerID="26e605795fdfd1c66aa2dbb35d2267a9fa777e6569065b2477f6e68ad7bc60c6" exitCode=60
Oct 30 20:15:03.136037 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:03.136024    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75" event={"ID":"a820cb88-eb76-459c-bb08-28837aa714a1","Type":"ContainerDied","Data":"26e605795fdfd1c66aa2dbb35d2267a9fa777e6569065b2477f6e68ad7bc60c6"}
Oct 30 20:15:03.136624 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:03.136512    5958 scope.go:122] "RemoveContainer" containerID="26e605795fdfd1c66aa2dbb35d2267a9fa777e6569065b2477f6e68ad7bc60c6"
Oct 30 20:15:03.139915 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:03.139877    5958 generic.go:356] "Generic (PLEG): container finished" podID="5c2103c1-53ee-4aaf-ab8e-75c5c93390a1" containerID="dddc898421a27b3c78cf2c024582a2af32647a48d69d3b6c9d25c63c4faed7c1" exitCode=60
Oct 30 20:15:03.140010 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:03.139933    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl" event={"ID":"5c2103c1-53ee-4aaf-ab8e-75c5c93390a1","Type":"ContainerDied","Data":"dddc898421a27b3c78cf2c024582a2af32647a48d69d3b6c9d25c63c4faed7c1"}
Oct 30 20:15:03.140304 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:03.140287    5958 scope.go:122] "RemoveContainer" containerID="dddc898421a27b3c78cf2c024582a2af32647a48d69d3b6c9d25c63c4faed7c1"
Oct 30 20:15:04.151708 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:04.151666    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9" event={"ID":"ed65872e-55ca-416c-bf1d-05e1bc0047b5","Type":"ContainerStarted","Data":"de6f8130288c34de46fab68a5ea09e32697dd6d89c70fd82f25bc7c89b017acb"}
Oct 30 20:15:04.154316 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:04.154290    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7" event={"ID":"ea2b023d-3aaf-47f7-8fa2-385d0ab430a5","Type":"ContainerStarted","Data":"fe3f0d94eff79c5ef68da61af3c898e3bffb676e5c51a9283e723faf04babbcd"}
Oct 30 20:15:04.156244 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:04.156220    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc" event={"ID":"b7fae60d-346d-4d4a-8851-82586d2ddf65","Type":"ContainerStarted","Data":"8659ced8bd5c373562477c7981fe5aa1e46018de924d5214b343f5b90eae0734"}
Oct 30 20:15:04.158342 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:04.158316    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75" event={"ID":"a820cb88-eb76-459c-bb08-28837aa714a1","Type":"ContainerStarted","Data":"6fb0b37c1a5e5f09dc59340064d1ad379ca703b6fc9c55325aefd27f745c7ee2"}
Oct 30 20:15:04.160439 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:04.160413    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl" event={"ID":"5c2103c1-53ee-4aaf-ab8e-75c5c93390a1","Type":"ContainerStarted","Data":"d764f21ee112551dca347ccd8bace8cda09f6450b49dfe2b966944b1f89d00c4"}
Oct 30 20:15:04.162297 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:04.162273    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c" event={"ID":"e20ab886-1287-453f-baea-c74e65e5ccad","Type":"ContainerStarted","Data":"b328d5cbdf34b3e08100ff6ad7dc2c3c3ec27c3438fb1472dccfd5b8e636d315"}
Oct 30 20:15:05.173703 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:05.173671    5958 generic.go:356] "Generic (PLEG): container finished" podID="ea2b023d-3aaf-47f7-8fa2-385d0ab430a5" containerID="fe3f0d94eff79c5ef68da61af3c898e3bffb676e5c51a9283e723faf04babbcd" exitCode=60
Oct 30 20:15:05.174168 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:05.173733    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7" event={"ID":"ea2b023d-3aaf-47f7-8fa2-385d0ab430a5","Type":"ContainerDied","Data":"fe3f0d94eff79c5ef68da61af3c898e3bffb676e5c51a9283e723faf04babbcd"}
Oct 30 20:15:05.174168 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:05.173822    5958 scope.go:122] "RemoveContainer" containerID="5c6c797779faa182aff840bd5454041048ae56878bc3aba2b96b3b04c63624f4"
Oct 30 20:15:05.174168 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:05.174078    5958 scope.go:122] "RemoveContainer" containerID="fe3f0d94eff79c5ef68da61af3c898e3bffb676e5c51a9283e723faf04babbcd"
Oct 30 20:15:05.174486 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:15:05.174436    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7_odh-ai-gateway-infra(ea2b023d-3aaf-47f7-8fa2-385d0ab430a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7" podUID="ea2b023d-3aaf-47f7-8fa2-385d0ab430a5"
Oct 30 20:15:05.175916 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:05.175893    5958 generic.go:356] "Generic (PLEG): container finished" podID="b7fae60d-346d-4d4a-8851-82586d2ddf65" containerID="8659ced8bd5c373562477c7981fe5aa1e46018de924d5214b343f5b90eae0734" exitCode=60
Oct 30 20:15:05.176020 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:05.175954    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc" event={"ID":"b7fae60d-346d-4d4a-8851-82586d2ddf65","Type":"ContainerDied","Data":"8659ced8bd5c373562477c7981fe5aa1e46018de924d5214b343f5b90eae0734"}
Oct 30 20:15:05.176212 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:05.176191    5958 scope.go:122] "RemoveContainer" containerID="8659ced8bd5c373562477c7981fe5aa1e46018de924d5214b343f5b90eae0734"
Oct 30 20:15:05.176558 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:15:05.176535    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc_odh-ai-gateway-infra(b7fae60d-346d-4d4a-8851-82586d2ddf65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc" podUID="b7fae60d-346d-4d4a-8851-82586d2ddf65"
Oct 30 20:15:05.178110 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:05.178089    5958 generic.go:356] "Generic (PLEG): container finished" podID="a820cb88-eb76-459c-bb08-28837aa714a1" containerID="6fb0b37c1a5e5f09dc59340064d1ad379ca703b6fc9c55325aefd27f745c7ee2" exitCode=60
Oct 30 20:15:05.178209 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:05.178167    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75" event={"ID":"a820cb88-eb76-459c-bb08-28837aa714a1","Type":"ContainerDied","Data":"6fb0b37c1a5e5f09dc59340064d1ad379ca703b6fc9c55325aefd27f745c7ee2"}
Oct 30 20:15:05.178390 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:05.178375    5958 scope.go:122] "RemoveContainer" containerID="6fb0b37c1a5e5f09dc59340064d1ad379ca703b6fc9c55325aefd27f745c7ee2"
Oct 30 20:15:05.178705 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:15:05.178680    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75_odh-ai-gateway-infra(a820cb88-eb76-459c-bb08-28837aa714a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75" podUID="a820cb88-eb76-459c-bb08-28837aa714a1"
Oct 30 20:15:05.180107 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:05.180088    5958 generic.go:356] "Generic (PLEG): container finished" podID="5c2103c1-53ee-4aaf-ab8e-75c5c93390a1" containerID="d764f21ee112551dca347ccd8bace8cda09f6450b49dfe2b966944b1f89d00c4" exitCode=60
Oct 30 20:15:05.180186 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:05.180150    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl" event={"ID":"5c2103c1-53ee-4aaf-ab8e-75c5c93390a1","Type":"ContainerDied","Data":"d764f21ee112551dca347ccd8bace8cda09f6450b49dfe2b966944b1f89d00c4"}
Oct 30 20:15:05.180372 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:05.180357    5958 scope.go:122] "RemoveContainer" containerID="d764f21ee112551dca347ccd8bace8cda09f6450b49dfe2b966944b1f89d00c4"
Oct 30 20:15:05.180688 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:15:05.180665    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl_odh-ai-gateway-infra(5c2103c1-53ee-4aaf-ab8e-75c5c93390a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl" podUID="5c2103c1-53ee-4aaf-ab8e-75c5c93390a1"
Oct 30 20:15:05.182213 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:05.182194    5958 generic.go:356] "Generic (PLEG): container finished" podID="e20ab886-1287-453f-baea-c74e65e5ccad" containerID="b328d5cbdf34b3e08100ff6ad7dc2c3c3ec27c3438fb1472dccfd5b8e636d315" exitCode=60
Oct 30 20:15:05.182287 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:05.182228    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c" event={"ID":"e20ab886-1287-453f-baea-c74e65e5ccad","Type":"ContainerDied","Data":"b328d5cbdf34b3e08100ff6ad7dc2c3c3ec27c3438fb1472dccfd5b8e636d315"}
Oct 30 20:15:05.182587 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:05.182567    5958 scope.go:122] "RemoveContainer" containerID="b328d5cbdf34b3e08100ff6ad7dc2c3c3ec27c3438fb1472dccfd5b8e636d315"
Oct 30 20:15:05.182916 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:15:05.182895    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-wr98c_odh-ai-gateway-infra(e20ab886-1287-453f-baea-c74e65e5ccad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c" podUID="e20ab886-1287-453f-baea-c74e65e5ccad"
Oct 30 20:15:05.184280 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:05.184256    5958 generic.go:356] "Generic (PLEG): container finished" podID="ed65872e-55ca-416c-bf1d-05e1bc0047b5" containerID="de6f8130288c34de46fab68a5ea09e32697dd6d89c70fd82f25bc7c89b017acb" exitCode=60
Oct 30 20:15:05.184380 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:05.184291    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9" event={"ID":"ed65872e-55ca-416c-bf1d-05e1bc0047b5","Type":"ContainerDied","Data":"de6f8130288c34de46fab68a5ea09e32697dd6d89c70fd82f25bc7c89b017acb"}
Oct 30 20:15:05.184545 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:05.184529    5958 scope.go:122] "RemoveContainer" containerID="de6f8130288c34de46fab68a5ea09e32697dd6d89c70fd82f25bc7c89b017acb"
Oct 30 20:15:05.184796 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:15:05.184776    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9_odh-ai-gateway-infra(ed65872e-55ca-416c-bf1d-05e1bc0047b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9" podUID="ed65872e-55ca-416c-bf1d-05e1bc0047b5"
Oct 30 20:15:05.188022 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:05.188005    5958 scope.go:122] "RemoveContainer" containerID="e13ebbc84c604989cf4f763cd4e7fe92d6988da0c57e2a957ad5b3b8cff41b4a"
Oct 30 20:15:05.199917 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:05.199897    5958 scope.go:122] "RemoveContainer" containerID="26e605795fdfd1c66aa2dbb35d2267a9fa777e6569065b2477f6e68ad7bc60c6"
Oct 30 20:15:05.211443 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:05.211423    5958 scope.go:122] "RemoveContainer" containerID="dddc898421a27b3c78cf2c024582a2af32647a48d69d3b6c9d25c63c4faed7c1"
Oct 30 20:15:05.224638 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:05.224613    5958 scope.go:122] "RemoveContainer" containerID="04b11d974e2e95756fb30c944d9e1c27040e98a1a8b63efdee4abccec6d8fed4"
Oct 30 20:15:05.236426 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:05.236397    5958 scope.go:122] "RemoveContainer" containerID="57b96604017eb994e0872642b31b4c75c2d15c466ed9ff9aec1ef397fb26862a"
Oct 30 20:15:06.194167 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:06.194141    5958 scope.go:122] "RemoveContainer" containerID="de6f8130288c34de46fab68a5ea09e32697dd6d89c70fd82f25bc7c89b017acb"
Oct 30 20:15:06.194610 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:15:06.194438    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9_odh-ai-gateway-infra(ed65872e-55ca-416c-bf1d-05e1bc0047b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9" podUID="ed65872e-55ca-416c-bf1d-05e1bc0047b5"
Oct 30 20:15:06.195618 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:06.195597    5958 scope.go:122] "RemoveContainer" containerID="fe3f0d94eff79c5ef68da61af3c898e3bffb676e5c51a9283e723faf04babbcd"
Oct 30 20:15:06.195894 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:15:06.195873    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7_odh-ai-gateway-infra(ea2b023d-3aaf-47f7-8fa2-385d0ab430a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7" podUID="ea2b023d-3aaf-47f7-8fa2-385d0ab430a5"
Oct 30 20:15:06.201953 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:06.201932    5958 scope.go:122] "RemoveContainer" containerID="8659ced8bd5c373562477c7981fe5aa1e46018de924d5214b343f5b90eae0734"
Oct 30 20:15:06.202229 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:15:06.202209    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc_odh-ai-gateway-infra(b7fae60d-346d-4d4a-8851-82586d2ddf65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc" podUID="b7fae60d-346d-4d4a-8851-82586d2ddf65"
Oct 30 20:15:06.203543 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:06.203523    5958 scope.go:122] "RemoveContainer" containerID="6fb0b37c1a5e5f09dc59340064d1ad379ca703b6fc9c55325aefd27f745c7ee2"
Oct 30 20:15:06.203771 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:15:06.203751    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75_odh-ai-gateway-infra(a820cb88-eb76-459c-bb08-28837aa714a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75" podUID="a820cb88-eb76-459c-bb08-28837aa714a1"
Oct 30 20:15:06.205059 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:06.205037    5958 scope.go:122] "RemoveContainer" containerID="d764f21ee112551dca347ccd8bace8cda09f6450b49dfe2b966944b1f89d00c4"
Oct 30 20:15:06.205347 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:15:06.205323    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl_odh-ai-gateway-infra(5c2103c1-53ee-4aaf-ab8e-75c5c93390a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl" podUID="5c2103c1-53ee-4aaf-ab8e-75c5c93390a1"
Oct 30 20:15:06.206656 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:06.206638    5958 scope.go:122] "RemoveContainer" containerID="b328d5cbdf34b3e08100ff6ad7dc2c3c3ec27c3438fb1472dccfd5b8e636d315"
Oct 30 20:15:06.206873 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:15:06.206856    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-wr98c_odh-ai-gateway-infra(e20ab886-1287-453f-baea-c74e65e5ccad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c" podUID="e20ab886-1287-453f-baea-c74e65e5ccad"
Oct 30 20:15:06.607903 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:15:06.607864    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:15:10.470488 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:10.470412    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5"]
Oct 30 20:15:10.590176 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:10.590144    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z"]
Oct 30 20:15:15.403444 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:15.403417    5958 scope.go:122] "RemoveContainer" containerID="b328d5cbdf34b3e08100ff6ad7dc2c3c3ec27c3438fb1472dccfd5b8e636d315"
Oct 30 20:15:15.403889 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:15.403607    5958 scope.go:122] "RemoveContainer" containerID="fe3f0d94eff79c5ef68da61af3c898e3bffb676e5c51a9283e723faf04babbcd"
Oct 30 20:15:15.403889 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:15.403685    5958 scope.go:122] "RemoveContainer" containerID="6fb0b37c1a5e5f09dc59340064d1ad379ca703b6fc9c55325aefd27f745c7ee2"
Oct 30 20:15:15.470830 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:15.470778    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5" podUID="20541f8a-ed94-4723-adc8-f6ee759ca566" containerName="payload-processing" containerID="cri-o://49c75723d7d7da8e471ca208e4dbe0a24db0e5d19265c6ca953a27e04506c239" gracePeriod=25
Oct 30 20:15:15.591402 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:15.591354    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z" podUID="f03388cb-31a5-41c1-9997-116ec2be50d9" containerName="payload-pre-processing" containerID="cri-o://29c23d1b1ae9f14513cd498a9094827ae996fd4fa9ebf1c648e1ceb78bbde88c" gracePeriod=25
Oct 30 20:15:15.963674 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:15.963616    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z"
Oct 30 20:15:16.003241 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.003218    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f03388cb-31a5-41c1-9997-116ec2be50d9-kube-api-access-zp6c9\" (UniqueName: \"kubernetes.io/projected/f03388cb-31a5-41c1-9997-116ec2be50d9-kube-api-access-zp6c9\") pod \"f03388cb-31a5-41c1-9997-116ec2be50d9\" (UID: \"f03388cb-31a5-41c1-9997-116ec2be50d9\") "
Oct 30 20:15:16.003400 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.003385    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f03388cb-31a5-41c1-9997-116ec2be50d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03388cb-31a5-41c1-9997-116ec2be50d9-plugins-config-volume\") pod \"f03388cb-31a5-41c1-9997-116ec2be50d9\" (UID: \"f03388cb-31a5-41c1-9997-116ec2be50d9\") "
Oct 30 20:15:16.003825 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.003767    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f03388cb-31a5-41c1-9997-116ec2be50d9-plugins-config-volume" pod "f03388cb-31a5-41c1-9997-116ec2be50d9" (UID: "f03388cb-31a5-41c1-9997-116ec2be50d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:16.004199 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.004181    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03388cb-31a5-41c1-9997-116ec2be50d9-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:16.036440 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.036402    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03388cb-31a5-41c1-9997-116ec2be50d9-kube-api-access-zp6c9" pod "f03388cb-31a5-41c1-9997-116ec2be50d9" (UID: "f03388cb-31a5-41c1-9997-116ec2be50d9"). InnerVolumeSpecName "kube-api-access-zp6c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:16.055642 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.055619    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5"
Oct 30 20:15:16.105622 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.105591    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20541f8a-ed94-4723-adc8-f6ee759ca566-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20541f8a-ed94-4723-adc8-f6ee759ca566-plugins-config-volume\") pod \"20541f8a-ed94-4723-adc8-f6ee759ca566\" (UID: \"20541f8a-ed94-4723-adc8-f6ee759ca566\") "
Oct 30 20:15:16.105790 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.105773    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20541f8a-ed94-4723-adc8-f6ee759ca566-kube-api-access-bzw82\" (UniqueName: \"kubernetes.io/projected/20541f8a-ed94-4723-adc8-f6ee759ca566-kube-api-access-bzw82\") pod \"20541f8a-ed94-4723-adc8-f6ee759ca566\" (UID: \"20541f8a-ed94-4723-adc8-f6ee759ca566\") "
Oct 30 20:15:16.106034 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.106006    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20541f8a-ed94-4723-adc8-f6ee759ca566-plugins-config-volume" pod "20541f8a-ed94-4723-adc8-f6ee759ca566" (UID: "20541f8a-ed94-4723-adc8-f6ee759ca566"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:16.106541 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.106520    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20541f8a-ed94-4723-adc8-f6ee759ca566-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:16.106627 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.106544    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp6c9\" (UniqueName: \"kubernetes.io/projected/f03388cb-31a5-41c1-9997-116ec2be50d9-kube-api-access-zp6c9\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:16.108621 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.108595    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20541f8a-ed94-4723-adc8-f6ee759ca566-kube-api-access-bzw82" pod "20541f8a-ed94-4723-adc8-f6ee759ca566" (UID: "20541f8a-ed94-4723-adc8-f6ee759ca566"). InnerVolumeSpecName "kube-api-access-bzw82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:16.208346 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.208317    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzw82\" (UniqueName: \"kubernetes.io/projected/20541f8a-ed94-4723-adc8-f6ee759ca566-kube-api-access-bzw82\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:16.340173 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.340143    5958 generic.go:356] "Generic (PLEG): container finished" podID="20541f8a-ed94-4723-adc8-f6ee759ca566" containerID="49c75723d7d7da8e471ca208e4dbe0a24db0e5d19265c6ca953a27e04506c239" exitCode=0
Oct 30 20:15:16.340351 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.340193    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5"
Oct 30 20:15:16.340351 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.340230    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5" event={"ID":"20541f8a-ed94-4723-adc8-f6ee759ca566","Type":"ContainerDied","Data":"49c75723d7d7da8e471ca208e4dbe0a24db0e5d19265c6ca953a27e04506c239"}
Oct 30 20:15:16.340351 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.340283    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5" event={"ID":"20541f8a-ed94-4723-adc8-f6ee759ca566","Type":"ContainerDied","Data":"025f31942ad571fb2275cd870dd589c0e766b6546b3406723be1b445acd8949f"}
Oct 30 20:15:16.340351 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.340329    5958 scope.go:122] "RemoveContainer" containerID="49c75723d7d7da8e471ca208e4dbe0a24db0e5d19265c6ca953a27e04506c239"
Oct 30 20:15:16.342811 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.342786    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75" event={"ID":"a820cb88-eb76-459c-bb08-28837aa714a1","Type":"ContainerStarted","Data":"d85c5c1233b0f836e4a9cdf5d44a0390e6187cf99dae4d5c84ee44c75dcc3188"}
Oct 30 20:15:16.345188 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.345161    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c" event={"ID":"e20ab886-1287-453f-baea-c74e65e5ccad","Type":"ContainerStarted","Data":"c84bc9e10b7b5d3bae8c353134bbba41b42183e89ee8339671026cbfc53ead96"}
Oct 30 20:15:16.347046 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.347023    5958 generic.go:356] "Generic (PLEG): container finished" podID="f03388cb-31a5-41c1-9997-116ec2be50d9" containerID="29c23d1b1ae9f14513cd498a9094827ae996fd4fa9ebf1c648e1ceb78bbde88c" exitCode=0
Oct 30 20:15:16.347176 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.347072    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z"
Oct 30 20:15:16.347176 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.347067    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z" event={"ID":"f03388cb-31a5-41c1-9997-116ec2be50d9","Type":"ContainerDied","Data":"29c23d1b1ae9f14513cd498a9094827ae996fd4fa9ebf1c648e1ceb78bbde88c"}
Oct 30 20:15:16.347299 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.347184    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z" event={"ID":"f03388cb-31a5-41c1-9997-116ec2be50d9","Type":"ContainerDied","Data":"4693a0ca3d937d5e21dbd57ae6125f85c2e5cb0c8819ff6b4e592367afd80ec9"}
Oct 30 20:15:16.352699 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.352681    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7" event={"ID":"ea2b023d-3aaf-47f7-8fa2-385d0ab430a5","Type":"ContainerStarted","Data":"43168993eb60e4586a76819ed19398b61c35c708345810f108bee8f7089ee827"}
Oct 30 20:15:16.355062 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.355044    5958 scope.go:122] "RemoveContainer" containerID="49c75723d7d7da8e471ca208e4dbe0a24db0e5d19265c6ca953a27e04506c239"
Oct 30 20:15:16.355315 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:15:16.355297    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49c75723d7d7da8e471ca208e4dbe0a24db0e5d19265c6ca953a27e04506c239\": container with ID starting with 49c75723d7d7da8e471ca208e4dbe0a24db0e5d19265c6ca953a27e04506c239 not found: ID does not exist" containerID="49c75723d7d7da8e471ca208e4dbe0a24db0e5d19265c6ca953a27e04506c239"
Oct 30 20:15:16.355531 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.355334    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49c75723d7d7da8e471ca208e4dbe0a24db0e5d19265c6ca953a27e04506c239"} err="failed to get container status \"49c75723d7d7da8e471ca208e4dbe0a24db0e5d19265c6ca953a27e04506c239\": rpc error: code = NotFound desc = could not find container \"49c75723d7d7da8e471ca208e4dbe0a24db0e5d19265c6ca953a27e04506c239\": container with ID starting with 49c75723d7d7da8e471ca208e4dbe0a24db0e5d19265c6ca953a27e04506c239 not found: ID does not exist"
Oct 30 20:15:16.355531 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.355358    5958 scope.go:122] "RemoveContainer" containerID="29c23d1b1ae9f14513cd498a9094827ae996fd4fa9ebf1c648e1ceb78bbde88c"
Oct 30 20:15:16.365585 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.365566    5958 scope.go:122] "RemoveContainer" containerID="29c23d1b1ae9f14513cd498a9094827ae996fd4fa9ebf1c648e1ceb78bbde88c"
Oct 30 20:15:16.365831 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:15:16.365812    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29c23d1b1ae9f14513cd498a9094827ae996fd4fa9ebf1c648e1ceb78bbde88c\": container with ID starting with 29c23d1b1ae9f14513cd498a9094827ae996fd4fa9ebf1c648e1ceb78bbde88c not found: ID does not exist" containerID="29c23d1b1ae9f14513cd498a9094827ae996fd4fa9ebf1c648e1ceb78bbde88c"
Oct 30 20:15:16.365886 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.365840    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29c23d1b1ae9f14513cd498a9094827ae996fd4fa9ebf1c648e1ceb78bbde88c"} err="failed to get container status \"29c23d1b1ae9f14513cd498a9094827ae996fd4fa9ebf1c648e1ceb78bbde88c\": rpc error: code = NotFound desc = could not find container \"29c23d1b1ae9f14513cd498a9094827ae996fd4fa9ebf1c648e1ceb78bbde88c\": container with ID starting with 29c23d1b1ae9f14513cd498a9094827ae996fd4fa9ebf1c648e1ceb78bbde88c not found: ID does not exist"
Oct 30 20:15:16.508229 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.508200    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z"]
Oct 30 20:15:16.516545 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.516521    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-577fb85a-85cb849879-fk47z"]
Oct 30 20:15:16.979040 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.978972    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5"]
Oct 30 20:15:16.992366 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:16.992339    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-577fb85a-79666db8bd-mdlp5"]
Oct 30 20:15:17.362608 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:17.362573    5958 generic.go:356] "Generic (PLEG): container finished" podID="a820cb88-eb76-459c-bb08-28837aa714a1" containerID="d85c5c1233b0f836e4a9cdf5d44a0390e6187cf99dae4d5c84ee44c75dcc3188" exitCode=60
Oct 30 20:15:17.362781 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:17.362647    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75" event={"ID":"a820cb88-eb76-459c-bb08-28837aa714a1","Type":"ContainerDied","Data":"d85c5c1233b0f836e4a9cdf5d44a0390e6187cf99dae4d5c84ee44c75dcc3188"}
Oct 30 20:15:17.362781 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:17.362713    5958 scope.go:122] "RemoveContainer" containerID="6fb0b37c1a5e5f09dc59340064d1ad379ca703b6fc9c55325aefd27f745c7ee2"
Oct 30 20:15:17.363039 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:17.363022    5958 scope.go:122] "RemoveContainer" containerID="d85c5c1233b0f836e4a9cdf5d44a0390e6187cf99dae4d5c84ee44c75dcc3188"
Oct 30 20:15:17.363364 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:15:17.363336    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75_odh-ai-gateway-infra(a820cb88-eb76-459c-bb08-28837aa714a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75" podUID="a820cb88-eb76-459c-bb08-28837aa714a1"
Oct 30 20:15:17.365070 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:17.365046    5958 generic.go:356] "Generic (PLEG): container finished" podID="e20ab886-1287-453f-baea-c74e65e5ccad" containerID="c84bc9e10b7b5d3bae8c353134bbba41b42183e89ee8339671026cbfc53ead96" exitCode=60
Oct 30 20:15:17.365163 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:17.365075    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c" event={"ID":"e20ab886-1287-453f-baea-c74e65e5ccad","Type":"ContainerDied","Data":"c84bc9e10b7b5d3bae8c353134bbba41b42183e89ee8339671026cbfc53ead96"}
Oct 30 20:15:17.365401 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:17.365385    5958 scope.go:122] "RemoveContainer" containerID="c84bc9e10b7b5d3bae8c353134bbba41b42183e89ee8339671026cbfc53ead96"
Oct 30 20:15:17.365682 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:15:17.365662    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-wr98c_odh-ai-gateway-infra(e20ab886-1287-453f-baea-c74e65e5ccad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c" podUID="e20ab886-1287-453f-baea-c74e65e5ccad"
Oct 30 20:15:17.368180 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:17.368156    5958 generic.go:356] "Generic (PLEG): container finished" podID="ea2b023d-3aaf-47f7-8fa2-385d0ab430a5" containerID="43168993eb60e4586a76819ed19398b61c35c708345810f108bee8f7089ee827" exitCode=60
Oct 30 20:15:17.368289 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:17.368186    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7" event={"ID":"ea2b023d-3aaf-47f7-8fa2-385d0ab430a5","Type":"ContainerDied","Data":"43168993eb60e4586a76819ed19398b61c35c708345810f108bee8f7089ee827"}
Oct 30 20:15:17.368564 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:17.368546    5958 scope.go:122] "RemoveContainer" containerID="43168993eb60e4586a76819ed19398b61c35c708345810f108bee8f7089ee827"
Oct 30 20:15:17.368874 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:15:17.368851    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7_odh-ai-gateway-infra(ea2b023d-3aaf-47f7-8fa2-385d0ab430a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7" podUID="ea2b023d-3aaf-47f7-8fa2-385d0ab430a5"
Oct 30 20:15:17.377028 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:17.377008    5958 scope.go:122] "RemoveContainer" containerID="b328d5cbdf34b3e08100ff6ad7dc2c3c3ec27c3438fb1472dccfd5b8e636d315"
Oct 30 20:15:17.391068 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:17.391051    5958 scope.go:122] "RemoveContainer" containerID="fe3f0d94eff79c5ef68da61af3c898e3bffb676e5c51a9283e723faf04babbcd"
Oct 30 20:15:17.391593 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:17.391573    5958 scope.go:122] "RemoveContainer" containerID="de6f8130288c34de46fab68a5ea09e32697dd6d89c70fd82f25bc7c89b017acb"
Oct 30 20:15:17.391775 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:17.391753    5958 scope.go:122] "RemoveContainer" containerID="d764f21ee112551dca347ccd8bace8cda09f6450b49dfe2b966944b1f89d00c4"
Oct 30 20:15:17.403194 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:17.403169    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20541f8a-ed94-4723-adc8-f6ee759ca566" path="/var/lib/kubelet/pods/20541f8a-ed94-4723-adc8-f6ee759ca566/volumes"
Oct 30 20:15:17.403822 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:17.403800    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f03388cb-31a5-41c1-9997-116ec2be50d9" path="/var/lib/kubelet/pods/f03388cb-31a5-41c1-9997-116ec2be50d9/volumes"
Oct 30 20:15:17.645211 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:17.645115    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c"]
Oct 30 20:15:17.762109 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:17.762079    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75"]
Oct 30 20:15:17.878079 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:17.878046    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7"]
Oct 30 20:15:18.381652 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:18.381616    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl" event={"ID":"5c2103c1-53ee-4aaf-ab8e-75c5c93390a1","Type":"ContainerStarted","Data":"1146fa890d5517459f7a122977b3233b6058bfa5fc59e3ea05adcfe0249a9b58"}
Oct 30 20:15:18.385689 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:18.385659    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9" event={"ID":"ed65872e-55ca-416c-bf1d-05e1bc0047b5","Type":"ContainerStarted","Data":"f87489b2207607f460b83a6031805be29ee0b3be6885c942c19bf823d6877e18"}
Oct 30 20:15:18.390888 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:18.390863    5958 scope.go:122] "RemoveContainer" containerID="8659ced8bd5c373562477c7981fe5aa1e46018de924d5214b343f5b90eae0734"
Oct 30 20:15:18.691853 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:18.691829    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7"
Oct 30 20:15:18.695820 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:18.695801    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75"
Oct 30 20:15:18.699907 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:18.699891    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c"
Oct 30 20:15:18.859884 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:18.859860    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea2b023d-3aaf-47f7-8fa2-385d0ab430a5-kube-api-access-cs78q\" (UniqueName: \"kubernetes.io/projected/ea2b023d-3aaf-47f7-8fa2-385d0ab430a5-kube-api-access-cs78q\") pod \"ea2b023d-3aaf-47f7-8fa2-385d0ab430a5\" (UID: \"ea2b023d-3aaf-47f7-8fa2-385d0ab430a5\") "
Oct 30 20:15:18.860005 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:18.859982    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e20ab886-1287-453f-baea-c74e65e5ccad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e20ab886-1287-453f-baea-c74e65e5ccad-maas-api-service-ca\") pod \"e20ab886-1287-453f-baea-c74e65e5ccad\" (UID: \"e20ab886-1287-453f-baea-c74e65e5ccad\") "
Oct 30 20:15:18.860073 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:18.860065    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a820cb88-eb76-459c-bb08-28837aa714a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a820cb88-eb76-459c-bb08-28837aa714a1-maas-api-service-ca\") pod \"a820cb88-eb76-459c-bb08-28837aa714a1\" (UID: \"a820cb88-eb76-459c-bb08-28837aa714a1\") "
Oct 30 20:15:18.860168 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:18.860152    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a820cb88-eb76-459c-bb08-28837aa714a1-kube-api-access-mpss9\" (UniqueName: \"kubernetes.io/projected/a820cb88-eb76-459c-bb08-28837aa714a1-kube-api-access-mpss9\") pod \"a820cb88-eb76-459c-bb08-28837aa714a1\" (UID: \"a820cb88-eb76-459c-bb08-28837aa714a1\") "
Oct 30 20:15:18.860278 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:18.860256    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e20ab886-1287-453f-baea-c74e65e5ccad-kube-api-access-8c85z\" (UniqueName: \"kubernetes.io/projected/e20ab886-1287-453f-baea-c74e65e5ccad-kube-api-access-8c85z\") pod \"e20ab886-1287-453f-baea-c74e65e5ccad\" (UID: \"e20ab886-1287-453f-baea-c74e65e5ccad\") "
Oct 30 20:15:18.860409 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:18.860347    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e20ab886-1287-453f-baea-c74e65e5ccad-maas-api-service-ca" pod "e20ab886-1287-453f-baea-c74e65e5ccad" (UID: "e20ab886-1287-453f-baea-c74e65e5ccad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:18.860499 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:18.860435    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a820cb88-eb76-459c-bb08-28837aa714a1-maas-api-service-ca" pod "a820cb88-eb76-459c-bb08-28837aa714a1" (UID: "a820cb88-eb76-459c-bb08-28837aa714a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:18.860499 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:18.860445    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea2b023d-3aaf-47f7-8fa2-385d0ab430a5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2b023d-3aaf-47f7-8fa2-385d0ab430a5-maas-api-service-ca\") pod \"ea2b023d-3aaf-47f7-8fa2-385d0ab430a5\" (UID: \"ea2b023d-3aaf-47f7-8fa2-385d0ab430a5\") "
Oct 30 20:15:18.860757 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:18.860740    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2b023d-3aaf-47f7-8fa2-385d0ab430a5-maas-api-service-ca" pod "ea2b023d-3aaf-47f7-8fa2-385d0ab430a5" (UID: "ea2b023d-3aaf-47f7-8fa2-385d0ab430a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:18.861257 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:18.861237    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2b023d-3aaf-47f7-8fa2-385d0ab430a5-maas-api-service-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:18.861335 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:18.861262    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e20ab886-1287-453f-baea-c74e65e5ccad-maas-api-service-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:18.861335 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:18.861275    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a820cb88-eb76-459c-bb08-28837aa714a1-maas-api-service-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:18.862184 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:18.862162    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2b023d-3aaf-47f7-8fa2-385d0ab430a5-kube-api-access-cs78q" pod "ea2b023d-3aaf-47f7-8fa2-385d0ab430a5" (UID: "ea2b023d-3aaf-47f7-8fa2-385d0ab430a5"). InnerVolumeSpecName "kube-api-access-cs78q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:18.862267 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:18.862210    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a820cb88-eb76-459c-bb08-28837aa714a1-kube-api-access-mpss9" pod "a820cb88-eb76-459c-bb08-28837aa714a1" (UID: "a820cb88-eb76-459c-bb08-28837aa714a1"). InnerVolumeSpecName "kube-api-access-mpss9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:18.862519 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:18.862500    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20ab886-1287-453f-baea-c74e65e5ccad-kube-api-access-8c85z" pod "e20ab886-1287-453f-baea-c74e65e5ccad" (UID: "e20ab886-1287-453f-baea-c74e65e5ccad"). InnerVolumeSpecName "kube-api-access-8c85z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:18.963240 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:18.963163    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c85z\" (UniqueName: \"kubernetes.io/projected/e20ab886-1287-453f-baea-c74e65e5ccad-kube-api-access-8c85z\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:18.963240 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:18.963208    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs78q\" (UniqueName: \"kubernetes.io/projected/ea2b023d-3aaf-47f7-8fa2-385d0ab430a5-kube-api-access-cs78q\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:18.963240 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:18.963218    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpss9\" (UniqueName: \"kubernetes.io/projected/a820cb88-eb76-459c-bb08-28837aa714a1-kube-api-access-mpss9\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:19.400865 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:19.400837    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7"
Oct 30 20:15:19.402501 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:19.402422    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7" event={"ID":"ea2b023d-3aaf-47f7-8fa2-385d0ab430a5","Type":"ContainerDied","Data":"6e35a24f3f7f4e5a38f655f94dfb017d99e7cf9338e0b06615a4b170ddc53eb6"}
Oct 30 20:15:19.402501 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:19.402494    5958 scope.go:122] "RemoveContainer" containerID="43168993eb60e4586a76819ed19398b61c35c708345810f108bee8f7089ee827"
Oct 30 20:15:19.403943 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:19.403281    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc" event={"ID":"b7fae60d-346d-4d4a-8851-82586d2ddf65","Type":"ContainerStarted","Data":"251aa073c242ca078ce12995fdb09cb296474c1c838d867171c42738d049c5fa"}
Oct 30 20:15:19.405702 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:19.405669    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75"
Oct 30 20:15:19.408737 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:19.408709    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75" event={"ID":"a820cb88-eb76-459c-bb08-28837aa714a1","Type":"ContainerDied","Data":"3efec6e2ad0375b48558de473c917abbb8cd6441602b2dec29ce164be354e813"}
Oct 30 20:15:19.410683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:19.410644    5958 generic.go:356] "Generic (PLEG): container finished" podID="5c2103c1-53ee-4aaf-ab8e-75c5c93390a1" containerID="1146fa890d5517459f7a122977b3233b6058bfa5fc59e3ea05adcfe0249a9b58" exitCode=60
Oct 30 20:15:19.410817 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:19.410796    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl" event={"ID":"5c2103c1-53ee-4aaf-ab8e-75c5c93390a1","Type":"ContainerDied","Data":"1146fa890d5517459f7a122977b3233b6058bfa5fc59e3ea05adcfe0249a9b58"}
Oct 30 20:15:19.411147 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:19.411131    5958 scope.go:122] "RemoveContainer" containerID="1146fa890d5517459f7a122977b3233b6058bfa5fc59e3ea05adcfe0249a9b58"
Oct 30 20:15:19.411433 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:15:19.411411    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl_odh-ai-gateway-infra(5c2103c1-53ee-4aaf-ab8e-75c5c93390a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl" podUID="5c2103c1-53ee-4aaf-ab8e-75c5c93390a1"
Oct 30 20:15:19.412983 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:19.412961    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c"
Oct 30 20:15:19.413079 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:19.413018    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c" event={"ID":"e20ab886-1287-453f-baea-c74e65e5ccad","Type":"ContainerDied","Data":"3c15ec93803d27ba68c3f93947ec872fff589531a3ee1f9a24d2813208204aa3"}
Oct 30 20:15:19.415304 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:19.415283    5958 generic.go:356] "Generic (PLEG): container finished" podID="ed65872e-55ca-416c-bf1d-05e1bc0047b5" containerID="f87489b2207607f460b83a6031805be29ee0b3be6885c942c19bf823d6877e18" exitCode=60
Oct 30 20:15:19.415420 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:19.415333    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9" event={"ID":"ed65872e-55ca-416c-bf1d-05e1bc0047b5","Type":"ContainerDied","Data":"f87489b2207607f460b83a6031805be29ee0b3be6885c942c19bf823d6877e18"}
Oct 30 20:15:19.415744 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:19.415684    5958 scope.go:122] "RemoveContainer" containerID="f87489b2207607f460b83a6031805be29ee0b3be6885c942c19bf823d6877e18"
Oct 30 20:15:19.416003 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:15:19.415978    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9_odh-ai-gateway-infra(ed65872e-55ca-416c-bf1d-05e1bc0047b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9" podUID="ed65872e-55ca-416c-bf1d-05e1bc0047b5"
Oct 30 20:15:19.420768 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:19.420748    5958 scope.go:122] "RemoveContainer" containerID="d85c5c1233b0f836e4a9cdf5d44a0390e6187cf99dae4d5c84ee44c75dcc3188"
Oct 30 20:15:19.431629 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:19.431564    5958 scope.go:122] "RemoveContainer" containerID="d764f21ee112551dca347ccd8bace8cda09f6450b49dfe2b966944b1f89d00c4"
Oct 30 20:15:19.446489 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:19.446432    5958 scope.go:122] "RemoveContainer" containerID="c84bc9e10b7b5d3bae8c353134bbba41b42183e89ee8339671026cbfc53ead96"
Oct 30 20:15:19.456737 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:19.456716    5958 scope.go:122] "RemoveContainer" containerID="de6f8130288c34de46fab68a5ea09e32697dd6d89c70fd82f25bc7c89b017acb"
Oct 30 20:15:19.518788 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:19.518762    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl"]
Oct 30 20:15:19.635322 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:19.635295    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9"]
Oct 30 20:15:19.811563 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:19.811388    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c"]
Oct 30 20:15:19.818041 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:19.818003    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-wr98c"]
Oct 30 20:15:20.047180 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:20.047138    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75"]
Oct 30 20:15:20.055581 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:20.055550    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-779921-29889855-hnr75"]
Oct 30 20:15:20.166085 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:20.166059    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7"]
Oct 30 20:15:20.172733 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:20.172700    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-770dfe-29889855-kq8w7"]
Oct 30 20:15:20.433836 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:20.433800    5958 generic.go:356] "Generic (PLEG): container finished" podID="b7fae60d-346d-4d4a-8851-82586d2ddf65" containerID="251aa073c242ca078ce12995fdb09cb296474c1c838d867171c42738d049c5fa" exitCode=60
Oct 30 20:15:20.433975 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:20.433881    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc" event={"ID":"b7fae60d-346d-4d4a-8851-82586d2ddf65","Type":"ContainerDied","Data":"251aa073c242ca078ce12995fdb09cb296474c1c838d867171c42738d049c5fa"}
Oct 30 20:15:20.433975 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:20.433934    5958 scope.go:122] "RemoveContainer" containerID="8659ced8bd5c373562477c7981fe5aa1e46018de924d5214b343f5b90eae0734"
Oct 30 20:15:20.434409 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:20.434386    5958 scope.go:122] "RemoveContainer" containerID="251aa073c242ca078ce12995fdb09cb296474c1c838d867171c42738d049c5fa"
Oct 30 20:15:20.434843 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:15:20.434811    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc_odh-ai-gateway-infra(b7fae60d-346d-4d4a-8851-82586d2ddf65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc" podUID="b7fae60d-346d-4d4a-8851-82586d2ddf65"
Oct 30 20:15:20.720303 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:20.717883    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc"]
Oct 30 20:15:20.722756 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:20.722585    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl"
Oct 30 20:15:20.763329 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:20.763301    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9"
Oct 30 20:15:20.896628 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:20.896547    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed65872e-55ca-416c-bf1d-05e1bc0047b5-kube-api-access-cw6qn\" (UniqueName: \"kubernetes.io/projected/ed65872e-55ca-416c-bf1d-05e1bc0047b5-kube-api-access-cw6qn\") pod \"ed65872e-55ca-416c-bf1d-05e1bc0047b5\" (UID: \"ed65872e-55ca-416c-bf1d-05e1bc0047b5\") "
Oct 30 20:15:20.897068 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:20.896711    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c2103c1-53ee-4aaf-ab8e-75c5c93390a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2103c1-53ee-4aaf-ab8e-75c5c93390a1-maas-api-service-ca\") pod \"5c2103c1-53ee-4aaf-ab8e-75c5c93390a1\" (UID: \"5c2103c1-53ee-4aaf-ab8e-75c5c93390a1\") "
Oct 30 20:15:20.897068 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:20.896786    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed65872e-55ca-416c-bf1d-05e1bc0047b5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed65872e-55ca-416c-bf1d-05e1bc0047b5-maas-api-service-ca\") pod \"ed65872e-55ca-416c-bf1d-05e1bc0047b5\" (UID: \"ed65872e-55ca-416c-bf1d-05e1bc0047b5\") "
Oct 30 20:15:20.897068 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:20.897014    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c2103c1-53ee-4aaf-ab8e-75c5c93390a1-kube-api-access-25jml\" (UniqueName: \"kubernetes.io/projected/5c2103c1-53ee-4aaf-ab8e-75c5c93390a1-kube-api-access-25jml\") pod \"5c2103c1-53ee-4aaf-ab8e-75c5c93390a1\" (UID: \"5c2103c1-53ee-4aaf-ab8e-75c5c93390a1\") "
Oct 30 20:15:20.897289 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:20.897259    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2103c1-53ee-4aaf-ab8e-75c5c93390a1-maas-api-service-ca" pod "5c2103c1-53ee-4aaf-ab8e-75c5c93390a1" (UID: "5c2103c1-53ee-4aaf-ab8e-75c5c93390a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:20.897738 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:20.897706    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed65872e-55ca-416c-bf1d-05e1bc0047b5-maas-api-service-ca" pod "ed65872e-55ca-416c-bf1d-05e1bc0047b5" (UID: "ed65872e-55ca-416c-bf1d-05e1bc0047b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:20.898250 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:20.898228    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2103c1-53ee-4aaf-ab8e-75c5c93390a1-maas-api-service-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:20.898250 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:20.898250    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed65872e-55ca-416c-bf1d-05e1bc0047b5-maas-api-service-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:20.900119 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:20.900092    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed65872e-55ca-416c-bf1d-05e1bc0047b5-kube-api-access-cw6qn" pod "ed65872e-55ca-416c-bf1d-05e1bc0047b5" (UID: "ed65872e-55ca-416c-bf1d-05e1bc0047b5"). InnerVolumeSpecName "kube-api-access-cw6qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:20.900230 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:20.900163    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2103c1-53ee-4aaf-ab8e-75c5c93390a1-kube-api-access-25jml" pod "5c2103c1-53ee-4aaf-ab8e-75c5c93390a1" (UID: "5c2103c1-53ee-4aaf-ab8e-75c5c93390a1"). InnerVolumeSpecName "kube-api-access-25jml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:21.000905 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:21.000814    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25jml\" (UniqueName: \"kubernetes.io/projected/5c2103c1-53ee-4aaf-ab8e-75c5c93390a1-kube-api-access-25jml\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:21.000905 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:21.000855    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw6qn\" (UniqueName: \"kubernetes.io/projected/ed65872e-55ca-416c-bf1d-05e1bc0047b5-kube-api-access-cw6qn\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:21.404133 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:21.404101    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a820cb88-eb76-459c-bb08-28837aa714a1" path="/var/lib/kubelet/pods/a820cb88-eb76-459c-bb08-28837aa714a1/volumes"
Oct 30 20:15:21.404642 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:21.404627    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e20ab886-1287-453f-baea-c74e65e5ccad" path="/var/lib/kubelet/pods/e20ab886-1287-453f-baea-c74e65e5ccad/volumes"
Oct 30 20:15:21.405039 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:21.405026    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea2b023d-3aaf-47f7-8fa2-385d0ab430a5" path="/var/lib/kubelet/pods/ea2b023d-3aaf-47f7-8fa2-385d0ab430a5/volumes"
Oct 30 20:15:21.442575 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:21.442555    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9"
Oct 30 20:15:21.442697 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:21.442567    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9" event={"ID":"ed65872e-55ca-416c-bf1d-05e1bc0047b5","Type":"ContainerDied","Data":"a3efdfbe699985b660a10ecd5c32e0a807081d3d5c411e540b0a2b3cda0b1e03"}
Oct 30 20:15:21.442697 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:21.442621    5958 scope.go:122] "RemoveContainer" containerID="f87489b2207607f460b83a6031805be29ee0b3be6885c942c19bf823d6877e18"
Oct 30 20:15:21.447441 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:21.447423    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl"
Oct 30 20:15:21.447564 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:21.447435    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl" event={"ID":"5c2103c1-53ee-4aaf-ab8e-75c5c93390a1","Type":"ContainerDied","Data":"f8e7567046060abfecaecfbea8916e37f08bb6d64e1359f1b22c316164eb76e9"}
Oct 30 20:15:21.454478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:21.454431    5958 scope.go:122] "RemoveContainer" containerID="1146fa890d5517459f7a122977b3233b6058bfa5fc59e3ea05adcfe0249a9b58"
Oct 30 20:15:21.599825 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:21.599802    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc"
Oct 30 20:15:21.611226 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:21.611195    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl"]
Oct 30 20:15:21.617439 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:21.617417    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-37732de7-29889855-phcpl"]
Oct 30 20:15:21.711437 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:21.711385    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7fae60d-346d-4d4a-8851-82586d2ddf65-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7fae60d-346d-4d4a-8851-82586d2ddf65-maas-api-service-ca\") pod \"b7fae60d-346d-4d4a-8851-82586d2ddf65\" (UID: \"b7fae60d-346d-4d4a-8851-82586d2ddf65\") "
Oct 30 20:15:21.711437 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:21.711420    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7fae60d-346d-4d4a-8851-82586d2ddf65-kube-api-access-zdxbn\" (UniqueName: \"kubernetes.io/projected/b7fae60d-346d-4d4a-8851-82586d2ddf65-kube-api-access-zdxbn\") pod \"b7fae60d-346d-4d4a-8851-82586d2ddf65\" (UID: \"b7fae60d-346d-4d4a-8851-82586d2ddf65\") "
Oct 30 20:15:21.711795 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:21.711773    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7fae60d-346d-4d4a-8851-82586d2ddf65-maas-api-service-ca" pod "b7fae60d-346d-4d4a-8851-82586d2ddf65" (UID: "b7fae60d-346d-4d4a-8851-82586d2ddf65"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:21.712240 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:21.712222    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7fae60d-346d-4d4a-8851-82586d2ddf65-maas-api-service-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:21.713769 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:21.713747    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7fae60d-346d-4d4a-8851-82586d2ddf65-kube-api-access-zdxbn" pod "b7fae60d-346d-4d4a-8851-82586d2ddf65" (UID: "b7fae60d-346d-4d4a-8851-82586d2ddf65"). InnerVolumeSpecName "kube-api-access-zdxbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:21.727691 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:21.727670    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9"]
Oct 30 20:15:21.739274 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:21.739254    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-f3736ef9-29889855-g8vx9"]
Oct 30 20:15:21.814077 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:21.814047    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdxbn\" (UniqueName: \"kubernetes.io/projected/b7fae60d-346d-4d4a-8851-82586d2ddf65-kube-api-access-zdxbn\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:22.272844 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:22.272803    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"]
Oct 30 20:15:22.273357 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:22.273057    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g" podUID="0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2" containerName="istio-proxy" containerID="cri-o://35b0a1d334ce37916123a56d716f7f16a0d2e915c615021be2f2451bf43cd9e9" gracePeriod=30
Oct 30 20:15:22.321580 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:22.321535    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"]
Oct 30 20:15:22.321824 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:22.321794    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5" podUID="0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9" containerName="istio-proxy" containerID="cri-o://c90fd6a988ed86fdd24c7455be1733b2418ab0d8aa3aaa0b9f94ba8b58b3f36f" gracePeriod=30
Oct 30 20:15:22.325655 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:22.325629    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"]
Oct 30 20:15:22.325900 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:22.325876    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc" podUID="76eb3bb9-b9b8-4804-a31b-18165a69ce7f" containerName="istio-proxy" containerID="cri-o://b46e13efa9a4b44f447a159a054782d094611c181ae0558bdd5688e70ca25d6b" gracePeriod=30
Oct 30 20:15:22.329132 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:22.329100    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"]
Oct 30 20:15:22.329289 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:22.329269    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn" podUID="e8847b33-95a0-4136-9d77-82b9008d9d7d" containerName="istio-proxy" containerID="cri-o://53dbae9d1c4b4e7e6b70cf30f5a02413cbbed177020c682b402883e3a232be70" gracePeriod=30
Oct 30 20:15:22.463220 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:22.463193    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc"
Oct 30 20:15:22.463365 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:22.463192    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc" event={"ID":"b7fae60d-346d-4d4a-8851-82586d2ddf65","Type":"ContainerDied","Data":"07e979db704832a57e4c316c54e45f0563754ad0cac8e3e46b9e8bf704722bcb"}
Oct 30 20:15:22.463405 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:22.463383    5958 scope.go:122] "RemoveContainer" containerID="251aa073c242ca078ce12995fdb09cb296474c1c838d867171c42738d049c5fa"
Oct 30 20:15:22.881150 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:22.880791    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc"]
Oct 30 20:15:22.885172 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:22.885142    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f84d85-29889855-72vkc"]
Oct 30 20:15:23.401434 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:23.401396    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c2103c1-53ee-4aaf-ab8e-75c5c93390a1" path="/var/lib/kubelet/pods/5c2103c1-53ee-4aaf-ab8e-75c5c93390a1/volumes"
Oct 30 20:15:23.401944 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:23.401927    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7fae60d-346d-4d4a-8851-82586d2ddf65" path="/var/lib/kubelet/pods/b7fae60d-346d-4d4a-8851-82586d2ddf65/volumes"
Oct 30 20:15:23.402394 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:23.402379    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed65872e-55ca-416c-bf1d-05e1bc0047b5" path="/var/lib/kubelet/pods/ed65872e-55ca-416c-bf1d-05e1bc0047b5/volumes"
Oct 30 20:15:24.704709 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:24.704679    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:15:24.705065 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:24.704744    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc" podUID="76eb3bb9-b9b8-4804-a31b-18165a69ce7f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:15:26.428504 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:26.428440    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:15:26.428951 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:26.428536    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g" podUID="0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:15:27.367598 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.367567    5958 patch_prober.go:29] interesting pod/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 30 20:15:27.367721 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.367630    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn" podUID="e8847b33-95a0-4136-9d77-82b9008d9d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 30 20:15:27.538799 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.538632    5958 generic.go:356] "Generic (PLEG): container finished" podID="e8847b33-95a0-4136-9d77-82b9008d9d7d" containerID="53dbae9d1c4b4e7e6b70cf30f5a02413cbbed177020c682b402883e3a232be70" exitCode=0
Oct 30 20:15:27.538799 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.538735    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn" event={"ID":"e8847b33-95a0-4136-9d77-82b9008d9d7d","Type":"ContainerDied","Data":"53dbae9d1c4b4e7e6b70cf30f5a02413cbbed177020c682b402883e3a232be70"}
Oct 30 20:15:27.540832 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.540711    5958 generic.go:356] "Generic (PLEG): container finished" podID="76eb3bb9-b9b8-4804-a31b-18165a69ce7f" containerID="b46e13efa9a4b44f447a159a054782d094611c181ae0558bdd5688e70ca25d6b" exitCode=0
Oct 30 20:15:27.540832 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.540781    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc" event={"ID":"76eb3bb9-b9b8-4804-a31b-18165a69ce7f","Type":"ContainerDied","Data":"b46e13efa9a4b44f447a159a054782d094611c181ae0558bdd5688e70ca25d6b"}
Oct 30 20:15:27.542593 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.542429    5958 generic.go:356] "Generic (PLEG): container finished" podID="0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9" containerID="c90fd6a988ed86fdd24c7455be1733b2418ab0d8aa3aaa0b9f94ba8b58b3f36f" exitCode=0
Oct 30 20:15:27.542694 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.542591    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5" event={"ID":"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9","Type":"ContainerDied","Data":"c90fd6a988ed86fdd24c7455be1733b2418ab0d8aa3aaa0b9f94ba8b58b3f36f"}
Oct 30 20:15:27.544547 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.544425    5958 generic.go:356] "Generic (PLEG): container finished" podID="0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2" containerID="35b0a1d334ce37916123a56d716f7f16a0d2e915c615021be2f2451bf43cd9e9" exitCode=0
Oct 30 20:15:27.544547 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.544476    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g" event={"ID":"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2","Type":"ContainerDied","Data":"35b0a1d334ce37916123a56d716f7f16a0d2e915c615021be2f2451bf43cd9e9"}
Oct 30 20:15:27.768715 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.768691    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:15:27.772866 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.772842    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:15:27.777598 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.777581    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:15:27.781754 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.781736    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:15:27.845668 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.845640    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8847b33-95a0-4136-9d77-82b9008d9d7d-kube-api-access-hxfwf\" (UniqueName: \"kubernetes.io/projected/e8847b33-95a0-4136-9d77-82b9008d9d7d-kube-api-access-hxfwf\") pod \"e8847b33-95a0-4136-9d77-82b9008d9d7d\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") "
Oct 30 20:15:27.845815 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.845688    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-token\" (UniqueName: \"kubernetes.io/projected/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-token\") pod \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") "
Oct 30 20:15:27.845815 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.845725    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-workload-socket\") pod \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") "
Oct 30 20:15:27.845815 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.845768    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-workload-certs\") pod \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") "
Oct 30 20:15:27.845815 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.845793    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-token\" (UniqueName: \"kubernetes.io/projected/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-token\") pod \"e8847b33-95a0-4136-9d77-82b9008d9d7d\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") "
Oct 30 20:15:27.846032 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.845836    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istiod-ca-cert\") pod \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") "
Oct 30 20:15:27.846032 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.845887    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-podinfo\") pod \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") "
Oct 30 20:15:27.846032 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.845938    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-kube-api-access-hl82s\" (UniqueName: \"kubernetes.io/projected/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-kube-api-access-hl82s\") pod \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") "
Oct 30 20:15:27.846032 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.845955    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-podinfo\") pod \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") "
Oct 30 20:15:27.846032 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.845973    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-workload-certs\") pod \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") "
Oct 30 20:15:27.846032 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.846029    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-envoy\") pod \"e8847b33-95a0-4136-9d77-82b9008d9d7d\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") "
Oct 30 20:15:27.846313 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.846065    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-envoy\") pod \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") "
Oct 30 20:15:27.846313 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.846095    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-data\") pod \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") "
Oct 30 20:15:27.846313 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.846132    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-credential-socket\") pod \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") "
Oct 30 20:15:27.846313 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.846164    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-data\") pod \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") "
Oct 30 20:15:27.846313 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.846205    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istiod-ca-cert\") pod \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") "
Oct 30 20:15:27.846313 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.846254    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-kube-api-access-86hxt\" (UniqueName: \"kubernetes.io/projected/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-kube-api-access-86hxt\") pod \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") "
Oct 30 20:15:27.846313 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.846288    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-data\") pod \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") "
Oct 30 20:15:27.846662 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.846339    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-workload-socket\") pod \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") "
Oct 30 20:15:27.846662 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.846376    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-credential-socket\") pod \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") "
Oct 30 20:15:27.846662 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.846393    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-workload-socket" pod "0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2" (UID: "0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:27.846662 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.846477    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istiod-ca-cert\") pod \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") "
Oct 30 20:15:27.846662 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.846524    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-envoy\") pod \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\" (UID: \"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2\") "
Oct 30 20:15:27.846662 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.846578    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8847b33-95a0-4136-9d77-82b9008d9d7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8847b33-95a0-4136-9d77-82b9008d9d7d-istiod-ca-cert\") pod \"e8847b33-95a0-4136-9d77-82b9008d9d7d\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") "
Oct 30 20:15:27.846662 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.846624    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-workload-certs\") pod \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") "
Oct 30 20:15:27.846662 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.846652    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-workload-certs\") pod \"e8847b33-95a0-4136-9d77-82b9008d9d7d\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") "
Oct 30 20:15:27.847032 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.846706    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-data\") pod \"e8847b33-95a0-4136-9d77-82b9008d9d7d\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") "
Oct 30 20:15:27.847032 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.846759    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-podinfo\") pod \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") "
Oct 30 20:15:27.847032 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.846800    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-workload-socket\") pod \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") "
Oct 30 20:15:27.847032 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.846832    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-credential-socket\") pod \"e8847b33-95a0-4136-9d77-82b9008d9d7d\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") "
Oct 30 20:15:27.847032 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.846876    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-envoy\") pod \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") "
Oct 30 20:15:27.847032 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.846903    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-token\" (UniqueName: \"kubernetes.io/projected/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-token\") pod \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") "
Oct 30 20:15:27.847032 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.846961    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-credential-socket\") pod \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\" (UID: \"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9\") "
Oct 30 20:15:27.847032 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.847027    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-token\" (UniqueName: \"kubernetes.io/projected/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-token\") pod \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") "
Oct 30 20:15:27.847400 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.847147    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-kube-api-access-w5287\" (UniqueName: \"kubernetes.io/projected/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-kube-api-access-w5287\") pod \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\" (UID: \"76eb3bb9-b9b8-4804-a31b-18165a69ce7f\") "
Oct 30 20:15:27.847400 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.847313    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-workload-socket\") pod \"e8847b33-95a0-4136-9d77-82b9008d9d7d\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") "
Oct 30 20:15:27.847515 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.847424    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-podinfo\") pod \"e8847b33-95a0-4136-9d77-82b9008d9d7d\" (UID: \"e8847b33-95a0-4136-9d77-82b9008d9d7d\") "
Oct 30 20:15:27.848638 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.848615    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.849156 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.849117    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istiod-ca-cert" pod "0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9" (UID: "0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:27.849838 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.849306    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-token" pod "e8847b33-95a0-4136-9d77-82b9008d9d7d" (UID: "e8847b33-95a0-4136-9d77-82b9008d9d7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:27.849838 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.849384    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-workload-certs" pod "0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2" (UID: "0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:27.849838 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.849417    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-token" pod "0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2" (UID: "0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:27.849838 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.849572    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-credential-socket" pod "e8847b33-95a0-4136-9d77-82b9008d9d7d" (UID: "e8847b33-95a0-4136-9d77-82b9008d9d7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:27.849838 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.849716    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-credential-socket" pod "0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9" (UID: "0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:27.850124 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.849843    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8847b33-95a0-4136-9d77-82b9008d9d7d-istiod-ca-cert" pod "e8847b33-95a0-4136-9d77-82b9008d9d7d" (UID: "e8847b33-95a0-4136-9d77-82b9008d9d7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:27.850124 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.850063    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-workload-certs" pod "0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9" (UID: "0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:27.851083 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.850224    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-workload-certs" pod "e8847b33-95a0-4136-9d77-82b9008d9d7d" (UID: "e8847b33-95a0-4136-9d77-82b9008d9d7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:27.851083 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.850371    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-data" pod "e8847b33-95a0-4136-9d77-82b9008d9d7d" (UID: "e8847b33-95a0-4136-9d77-82b9008d9d7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:27.852533 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.852505    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-token" pod "0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9" (UID: "0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:27.852621 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.852586    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-podinfo" pod "e8847b33-95a0-4136-9d77-82b9008d9d7d" (UID: "e8847b33-95a0-4136-9d77-82b9008d9d7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:15:27.852673 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.852644    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-kube-api-access-hl82s" pod "0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2" (UID: "0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2"). InnerVolumeSpecName "kube-api-access-hl82s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:27.852673 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.852662    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8847b33-95a0-4136-9d77-82b9008d9d7d-kube-api-access-hxfwf" pod "e8847b33-95a0-4136-9d77-82b9008d9d7d" (UID: "e8847b33-95a0-4136-9d77-82b9008d9d7d"). InnerVolumeSpecName "kube-api-access-hxfwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:27.852888 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.852869    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-data" pod "0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2" (UID: "0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:27.852941 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.852896    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-data" pod "0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9" (UID: "0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:27.853238 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.853215    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istiod-ca-cert" pod "76eb3bb9-b9b8-4804-a31b-18165a69ce7f" (UID: "76eb3bb9-b9b8-4804-a31b-18165a69ce7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:27.853313 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.853292    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-token" pod "76eb3bb9-b9b8-4804-a31b-18165a69ce7f" (UID: "76eb3bb9-b9b8-4804-a31b-18165a69ce7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:27.853819 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.853794    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-envoy" pod "0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9" (UID: "0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:27.854063 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.854041    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-credential-socket" pod "76eb3bb9-b9b8-4804-a31b-18165a69ce7f" (UID: "76eb3bb9-b9b8-4804-a31b-18165a69ce7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:27.854241 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.854219    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-workload-socket" pod "76eb3bb9-b9b8-4804-a31b-18165a69ce7f" (UID: "76eb3bb9-b9b8-4804-a31b-18165a69ce7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:27.855237 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.854899    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-podinfo" pod "0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2" (UID: "0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:15:27.855237 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.854991    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-envoy" pod "0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2" (UID: "0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:27.855410 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.855273    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-podinfo" pod "0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9" (UID: "0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:15:27.855733 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.855537    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-workload-socket" pod "0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9" (UID: "0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:27.855733 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.855574    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istiod-ca-cert" pod "0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2" (UID: "0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:27.855883 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.855779    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-kube-api-access-86hxt" pod "0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9" (UID: "0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9"). InnerVolumeSpecName "kube-api-access-86hxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:27.855883 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.855819    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-data" pod "76eb3bb9-b9b8-4804-a31b-18165a69ce7f" (UID: "76eb3bb9-b9b8-4804-a31b-18165a69ce7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:27.856076 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.856025    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-credential-socket" pod "0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2" (UID: "0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:27.856076 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.856042    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-workload-certs" pod "76eb3bb9-b9b8-4804-a31b-18165a69ce7f" (UID: "76eb3bb9-b9b8-4804-a31b-18165a69ce7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:27.856200 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.856088    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-workload-socket" pod "e8847b33-95a0-4136-9d77-82b9008d9d7d" (UID: "e8847b33-95a0-4136-9d77-82b9008d9d7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:27.856659 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.856636    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-kube-api-access-w5287" pod "76eb3bb9-b9b8-4804-a31b-18165a69ce7f" (UID: "76eb3bb9-b9b8-4804-a31b-18165a69ce7f"). InnerVolumeSpecName "kube-api-access-w5287". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:27.856857 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.856837    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-envoy" pod "76eb3bb9-b9b8-4804-a31b-18165a69ce7f" (UID: "76eb3bb9-b9b8-4804-a31b-18165a69ce7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:27.857913 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.857895    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-envoy" pod "e8847b33-95a0-4136-9d77-82b9008d9d7d" (UID: "e8847b33-95a0-4136-9d77-82b9008d9d7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:15:27.858065 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.858047    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-podinfo" pod "76eb3bb9-b9b8-4804-a31b-18165a69ce7f" (UID: "76eb3bb9-b9b8-4804-a31b-18165a69ce7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:15:27.950415 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950381    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950415 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950411    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxfwf\" (UniqueName: \"kubernetes.io/projected/e8847b33-95a0-4136-9d77-82b9008d9d7d-kube-api-access-hxfwf\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950415 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950422    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950430    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950439    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950468    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950481    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950494    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl82s\" (UniqueName: \"kubernetes.io/projected/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-kube-api-access-hl82s\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950503    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950511    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950520    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950528    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950537    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950545    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950552    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950560    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950571    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86hxt\" (UniqueName: \"kubernetes.io/projected/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-kube-api-access-86hxt\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950579    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950587    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950595    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950605    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950613    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950620    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8847b33-95a0-4136-9d77-82b9008d9d7d-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950628    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950636    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950644    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950652    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950660    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950669    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950678    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950687    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950695    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.950683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950704    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.952373 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950712    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5287\" (UniqueName: \"kubernetes.io/projected/76eb3bb9-b9b8-4804-a31b-18165a69ce7f-kube-api-access-w5287\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:27.952373 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:27.950720    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8847b33-95a0-4136-9d77-82b9008d9d7d-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:28.553777 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:28.553747    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"
Oct 30 20:15:28.554211 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:28.553786    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn" event={"ID":"e8847b33-95a0-4136-9d77-82b9008d9d7d","Type":"ContainerDied","Data":"047651553b45e24af711071a2cb8901b47bd4814fe3808828c3dce1ec5a5e499"}
Oct 30 20:15:28.554211 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:28.553853    5958 scope.go:122] "RemoveContainer" containerID="53dbae9d1c4b4e7e6b70cf30f5a02413cbbed177020c682b402883e3a232be70"
Oct 30 20:15:28.555305 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:28.555284    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc" event={"ID":"76eb3bb9-b9b8-4804-a31b-18165a69ce7f","Type":"ContainerDied","Data":"74c7dce248e04fb06b7ebedd02a6ffab77d0090c52084b5f99594f8510aa0071"}
Oct 30 20:15:28.555400 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:28.555374    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"
Oct 30 20:15:28.558295 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:28.558262    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"
Oct 30 20:15:28.558403 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:28.558305    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5" event={"ID":"0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9","Type":"ContainerDied","Data":"61edda50818f495bee3372183ecdb8f8096d2dbfaf7725c8c8926ce31c393a44"}
Oct 30 20:15:28.560280 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:28.560259    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"
Oct 30 20:15:28.560385 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:28.560300    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g" event={"ID":"0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2","Type":"ContainerDied","Data":"f2d54e570beec1a411a06c10d43bf5ea5b2dafe3a584ab4d851900f0de0d51f2"}
Oct 30 20:15:28.567264 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:28.567246    5958 scope.go:122] "RemoveContainer" containerID="b46e13efa9a4b44f447a159a054782d094611c181ae0558bdd5688e70ca25d6b"
Oct 30 20:15:28.583211 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:28.583185    5958 scope.go:122] "RemoveContainer" containerID="c90fd6a988ed86fdd24c7455be1733b2418ab0d8aa3aaa0b9f94ba8b58b3f36f"
Oct 30 20:15:28.601143 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:28.601120    5958 scope.go:122] "RemoveContainer" containerID="35b0a1d334ce37916123a56d716f7f16a0d2e915c615021be2f2451bf43cd9e9"
Oct 30 20:15:28.723434 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:28.723394    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"]
Oct 30 20:15:28.773152 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:28.773126    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-fql5g"]
Oct 30 20:15:28.843228 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:28.843171    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"]
Oct 30 20:15:28.860479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:28.860437    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-bw4k5"]
Oct 30 20:15:29.003668 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:29.003638    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"]
Oct 30 20:15:29.007139 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:29.007115    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-5bqbc"]
Oct 30 20:15:29.111304 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:29.111246    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"]
Oct 30 20:15:29.114583 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:29.114557    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bebb42-openshift-default-68899dc5-6r9cn"]
Oct 30 20:15:29.407738 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:29.407663    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2" path="/var/lib/kubelet/pods/0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2/volumes"
Oct 30 20:15:29.408160 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:29.408145    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9" path="/var/lib/kubelet/pods/0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9/volumes"
Oct 30 20:15:29.408605 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:29.408591    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76eb3bb9-b9b8-4804-a31b-18165a69ce7f" path="/var/lib/kubelet/pods/76eb3bb9-b9b8-4804-a31b-18165a69ce7f/volumes"
Oct 30 20:15:29.409345 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:29.409330    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8847b33-95a0-4136-9d77-82b9008d9d7d" path="/var/lib/kubelet/pods/e8847b33-95a0-4136-9d77-82b9008d9d7d/volumes"
Oct 30 20:15:36.606403 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:15:36.606366    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:15:42.001994 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:42.001963    5958 scope.go:122] "RemoveContainer" containerID="c872f87e1bdd33996116effcca380a1c61728157428749dd669172593573f3ba"
Oct 30 20:15:42.012735 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:42.012715    5958 scope.go:122] "RemoveContainer" containerID="359b5396893b6ac99aae26a70cedd3eae2191b01e7f6530a2ff5cba1e93a992b"
Oct 30 20:15:42.021926 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:42.021907    5958 scope.go:122] "RemoveContainer" containerID="11c794ea4aef0fe76a208212c9b8aae4ddb9a4cae0598f64eea2722335209d29"
Oct 30 20:15:42.030562 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:15:42.030546    5958 scope.go:122] "RemoveContainer" containerID="7a39d705719cc9f7013ca49b6a4ef2cd8eb9b42f48b0c26bbb95257d46741a2d"
Oct 30 20:16:00.911299 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:00.911267    5958 generic.go:356] "Generic (PLEG): container finished" podID="23b32875-225b-469d-8794-c47053a71ebc" containerID="ab3eb8f380d8b23cc73f3c5f1a7a37c6559614a6200a59a9635ebce44917ebb6" exitCode=0
Oct 30 20:16:00.911794 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:00.911339    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd" event={"ID":"23b32875-225b-469d-8794-c47053a71ebc","Type":"ContainerDied","Data":"ab3eb8f380d8b23cc73f3c5f1a7a37c6559614a6200a59a9635ebce44917ebb6"}
Oct 30 20:16:00.911839 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:00.911795    5958 scope.go:122] "RemoveContainer" containerID="ab3eb8f380d8b23cc73f3c5f1a7a37c6559614a6200a59a9635ebce44917ebb6"
Oct 30 20:16:00.913499 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:00.913435    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:16:00.913574 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:00.913541    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:16:00.913613 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:00.913579    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:16:01.921164 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:01.921121    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd" event={"ID":"23b32875-225b-469d-8794-c47053a71ebc","Type":"ContainerStarted","Data":"857e436ff122c97e535b830567b55a6436e9ab3a89cd8e56f9cc9c39539da25e"}
Oct 30 20:16:02.798475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:02.798355    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:16:02.824434 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:02.822904    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 20:16:02.824434 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:02.822970    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd" podUID="23b32875-225b-469d-8794-c47053a71ebc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 20:16:03.794585 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:03.794546    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 20:16:03.795030 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:03.794601    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd" podUID="23b32875-225b-469d-8794-c47053a71ebc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 20:16:04.794847 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:04.794813    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 20:16:04.795253 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:04.794864    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd" podUID="23b32875-225b-469d-8794-c47053a71ebc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 20:16:05.794955 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:05.794923    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 20:16:05.795409 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:05.794982    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd" podUID="23b32875-225b-469d-8794-c47053a71ebc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 20:16:06.609600 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:16:06.609568    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:16:06.794341 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:06.794305    5958 patch_prober.go:29] interesting pod/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Oct 30 20:16:06.794549 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:06.794365    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd" podUID="23b32875-225b-469d-8794-c47053a71ebc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Oct 30 20:16:07.799384 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:07.799351    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:16:07.799868 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:07.799854    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:16:07.800699 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:07.800670    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:16:16.596070 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:16.596031    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"]
Oct 30 20:16:16.596573 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:16.596260    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp" podUID="6a6506dc-92f6-46cd-8e96-030a02603d96" containerName="maas-api" containerID="cri-o://0382469cf03112471d31ab82ec28375f63e1040de168acf57edd8e855346aec9" gracePeriod=30
Oct 30 20:16:16.928687 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:16.928661    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"
Oct 30 20:16:16.966007 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:16.965972    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a6506dc-92f6-46cd-8e96-030a02603d96-kube-api-access-64vjn\" (UniqueName: \"kubernetes.io/projected/6a6506dc-92f6-46cd-8e96-030a02603d96-kube-api-access-64vjn\") pod \"6a6506dc-92f6-46cd-8e96-030a02603d96\" (UID: \"6a6506dc-92f6-46cd-8e96-030a02603d96\") "
Oct 30 20:16:16.966203 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:16.966049    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a6506dc-92f6-46cd-8e96-030a02603d96-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a6506dc-92f6-46cd-8e96-030a02603d96-openshift-service-ca\") pod \"6a6506dc-92f6-46cd-8e96-030a02603d96\" (UID: \"6a6506dc-92f6-46cd-8e96-030a02603d96\") "
Oct 30 20:16:16.966203 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:16.966176    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a6506dc-92f6-46cd-8e96-030a02603d96-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a6506dc-92f6-46cd-8e96-030a02603d96-maas-api-tls\") pod \"6a6506dc-92f6-46cd-8e96-030a02603d96\" (UID: \"6a6506dc-92f6-46cd-8e96-030a02603d96\") "
Oct 30 20:16:16.966310 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:16.966297    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a6506dc-92f6-46cd-8e96-030a02603d96-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6506dc-92f6-46cd-8e96-030a02603d96-metrics-certs\") pod \"6a6506dc-92f6-46cd-8e96-030a02603d96\" (UID: \"6a6506dc-92f6-46cd-8e96-030a02603d96\") "
Oct 30 20:16:16.966435 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:16.966407    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a6506dc-92f6-46cd-8e96-030a02603d96-openshift-service-ca" pod "6a6506dc-92f6-46cd-8e96-030a02603d96" (UID: "6a6506dc-92f6-46cd-8e96-030a02603d96"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:16.967053 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:16.967026    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a6506dc-92f6-46cd-8e96-030a02603d96-openshift-service-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:16:16.968693 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:16.968670    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a6506dc-92f6-46cd-8e96-030a02603d96-metrics-certs" pod "6a6506dc-92f6-46cd-8e96-030a02603d96" (UID: "6a6506dc-92f6-46cd-8e96-030a02603d96"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:16.968904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:16.968888    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a6506dc-92f6-46cd-8e96-030a02603d96-maas-api-tls" pod "6a6506dc-92f6-46cd-8e96-030a02603d96" (UID: "6a6506dc-92f6-46cd-8e96-030a02603d96"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:16.968982 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:16.968958    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a6506dc-92f6-46cd-8e96-030a02603d96-kube-api-access-64vjn" pod "6a6506dc-92f6-46cd-8e96-030a02603d96" (UID: "6a6506dc-92f6-46cd-8e96-030a02603d96"). InnerVolumeSpecName "kube-api-access-64vjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:17.078517 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:17.069595    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64vjn\" (UniqueName: \"kubernetes.io/projected/6a6506dc-92f6-46cd-8e96-030a02603d96-kube-api-access-64vjn\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:16:17.078517 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:17.069637    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a6506dc-92f6-46cd-8e96-030a02603d96-maas-api-tls\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:16:17.078517 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:17.069652    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6506dc-92f6-46cd-8e96-030a02603d96-metrics-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:16:17.094428 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:17.093694    5958 generic.go:356] "Generic (PLEG): container finished" podID="6a6506dc-92f6-46cd-8e96-030a02603d96" containerID="0382469cf03112471d31ab82ec28375f63e1040de168acf57edd8e855346aec9" exitCode=0
Oct 30 20:16:17.094428 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:17.093872    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"
Oct 30 20:16:17.094428 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:17.093990    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp" event={"ID":"6a6506dc-92f6-46cd-8e96-030a02603d96","Type":"ContainerDied","Data":"0382469cf03112471d31ab82ec28375f63e1040de168acf57edd8e855346aec9"}
Oct 30 20:16:17.094428 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:17.094040    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp" event={"ID":"6a6506dc-92f6-46cd-8e96-030a02603d96","Type":"ContainerDied","Data":"6dafe36687fc50de21bf08009ff0867861aa88cb63bc22ed802756f18ff4570c"}
Oct 30 20:16:17.094428 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:17.094068    5958 scope.go:122] "RemoveContainer" containerID="0382469cf03112471d31ab82ec28375f63e1040de168acf57edd8e855346aec9"
Oct 30 20:16:17.128356 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:17.128332    5958 scope.go:122] "RemoveContainer" containerID="0382469cf03112471d31ab82ec28375f63e1040de168acf57edd8e855346aec9"
Oct 30 20:16:17.128745 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:16:17.128723    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0382469cf03112471d31ab82ec28375f63e1040de168acf57edd8e855346aec9\": container with ID starting with 0382469cf03112471d31ab82ec28375f63e1040de168acf57edd8e855346aec9 not found: ID does not exist" containerID="0382469cf03112471d31ab82ec28375f63e1040de168acf57edd8e855346aec9"
Oct 30 20:16:17.128839 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:17.128755    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0382469cf03112471d31ab82ec28375f63e1040de168acf57edd8e855346aec9"} err="failed to get container status \"0382469cf03112471d31ab82ec28375f63e1040de168acf57edd8e855346aec9\": rpc error: code = NotFound desc = could not find container \"0382469cf03112471d31ab82ec28375f63e1040de168acf57edd8e855346aec9\": container with ID starting with 0382469cf03112471d31ab82ec28375f63e1040de168acf57edd8e855346aec9 not found: ID does not exist"
Oct 30 20:16:17.267048 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:17.266955    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"]
Oct 30 20:16:17.271264 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:17.271232    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-577fb85a-86d6f9f769-q5pkp"]
Oct 30 20:16:17.401540 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:16:17.401508    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a6506dc-92f6-46cd-8e96-030a02603d96" path="/var/lib/kubelet/pods/6a6506dc-92f6-46cd-8e96-030a02603d96/volumes"
Oct 30 20:16:36.619631 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:16:36.619593    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:17:06.608900 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:17:06.608859    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:17:09.318257 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:09.318215    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn"]
Oct 30 20:17:14.318741 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:14.318695    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn" podUID="c35c95a6-d115-43cf-a332-19dc91c7bdd4" containerName="payload-pre-processing" containerID="cri-o://2226ed7b59e41b7dc00d12982a8192b3e8e556a3ffcdedaf4a3997784de659af" gracePeriod=25
Oct 30 20:17:14.625030 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:14.624993    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn"
Oct 30 20:17:14.676598 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:14.676566    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c35c95a6-d115-43cf-a332-19dc91c7bdd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35c95a6-d115-43cf-a332-19dc91c7bdd4-plugins-config-volume\") pod \"c35c95a6-d115-43cf-a332-19dc91c7bdd4\" (UID: \"c35c95a6-d115-43cf-a332-19dc91c7bdd4\") "
Oct 30 20:17:14.676775 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:14.676695    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c35c95a6-d115-43cf-a332-19dc91c7bdd4-kube-api-access-qwhrs\" (UniqueName: \"kubernetes.io/projected/c35c95a6-d115-43cf-a332-19dc91c7bdd4-kube-api-access-qwhrs\") pod \"c35c95a6-d115-43cf-a332-19dc91c7bdd4\" (UID: \"c35c95a6-d115-43cf-a332-19dc91c7bdd4\") "
Oct 30 20:17:14.677012 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:14.676989    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c35c95a6-d115-43cf-a332-19dc91c7bdd4-plugins-config-volume" pod "c35c95a6-d115-43cf-a332-19dc91c7bdd4" (UID: "c35c95a6-d115-43cf-a332-19dc91c7bdd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:14.677321 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:14.677303    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35c95a6-d115-43cf-a332-19dc91c7bdd4-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:14.679049 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:14.679014    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35c95a6-d115-43cf-a332-19dc91c7bdd4-kube-api-access-qwhrs" pod "c35c95a6-d115-43cf-a332-19dc91c7bdd4" (UID: "c35c95a6-d115-43cf-a332-19dc91c7bdd4"). InnerVolumeSpecName "kube-api-access-qwhrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:14.780170 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:14.780125    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwhrs\" (UniqueName: \"kubernetes.io/projected/c35c95a6-d115-43cf-a332-19dc91c7bdd4-kube-api-access-qwhrs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:14.801469 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:14.801411    5958 generic.go:356] "Generic (PLEG): container finished" podID="c35c95a6-d115-43cf-a332-19dc91c7bdd4" containerID="2226ed7b59e41b7dc00d12982a8192b3e8e556a3ffcdedaf4a3997784de659af" exitCode=0
Oct 30 20:17:14.801768 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:14.801538    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn"
Oct 30 20:17:14.802012 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:14.801533    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn" event={"ID":"c35c95a6-d115-43cf-a332-19dc91c7bdd4","Type":"ContainerDied","Data":"2226ed7b59e41b7dc00d12982a8192b3e8e556a3ffcdedaf4a3997784de659af"}
Oct 30 20:17:14.802158 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:14.802139    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn" event={"ID":"c35c95a6-d115-43cf-a332-19dc91c7bdd4","Type":"ContainerDied","Data":"632a52d96ab7d12ca467bf3d2636a77afac1b741e84658d64b30b9155afe01f2"}
Oct 30 20:17:14.802314 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:14.802268    5958 scope.go:122] "RemoveContainer" containerID="2226ed7b59e41b7dc00d12982a8192b3e8e556a3ffcdedaf4a3997784de659af"
Oct 30 20:17:14.824867 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:14.822681    5958 scope.go:122] "RemoveContainer" containerID="2226ed7b59e41b7dc00d12982a8192b3e8e556a3ffcdedaf4a3997784de659af"
Oct 30 20:17:14.824867 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:17:14.823293    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2226ed7b59e41b7dc00d12982a8192b3e8e556a3ffcdedaf4a3997784de659af\": container with ID starting with 2226ed7b59e41b7dc00d12982a8192b3e8e556a3ffcdedaf4a3997784de659af not found: ID does not exist" containerID="2226ed7b59e41b7dc00d12982a8192b3e8e556a3ffcdedaf4a3997784de659af"
Oct 30 20:17:14.824867 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:14.823336    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2226ed7b59e41b7dc00d12982a8192b3e8e556a3ffcdedaf4a3997784de659af"} err="failed to get container status \"2226ed7b59e41b7dc00d12982a8192b3e8e556a3ffcdedaf4a3997784de659af\": rpc error: code = NotFound desc = could not find container \"2226ed7b59e41b7dc00d12982a8192b3e8e556a3ffcdedaf4a3997784de659af\": container with ID starting with 2226ed7b59e41b7dc00d12982a8192b3e8e556a3ffcdedaf4a3997784de659af not found: ID does not exist"
Oct 30 20:17:14.972823 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:14.972786    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn"]
Oct 30 20:17:14.976269 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:14.976221    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7961ed-5c548fdcc5-c6jxn"]
Oct 30 20:17:15.401395 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:15.401347    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c35c95a6-d115-43cf-a332-19dc91c7bdd4" path="/var/lib/kubelet/pods/c35c95a6-d115-43cf-a332-19dc91c7bdd4/volumes"
Oct 30 20:17:23.967271 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:23.967177    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482"]
Oct 30 20:17:28.968603 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:28.968559    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482" podUID="73694a08-761d-412e-9a30-5c02be8900dd" containerName="payload-pre-processing" containerID="cri-o://0e208d30f988086800eea677204c765204521a97f9c60a5758271d3907d9b88b" gracePeriod=25
Oct 30 20:17:29.276251 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:29.276221    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482"
Oct 30 20:17:29.350735 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:29.350690    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73694a08-761d-412e-9a30-5c02be8900dd-kube-api-access-hmgr7\" (UniqueName: \"kubernetes.io/projected/73694a08-761d-412e-9a30-5c02be8900dd-kube-api-access-hmgr7\") pod \"73694a08-761d-412e-9a30-5c02be8900dd\" (UID: \"73694a08-761d-412e-9a30-5c02be8900dd\") "
Oct 30 20:17:29.355096 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:29.351182    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73694a08-761d-412e-9a30-5c02be8900dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73694a08-761d-412e-9a30-5c02be8900dd-plugins-config-volume\") pod \"73694a08-761d-412e-9a30-5c02be8900dd\" (UID: \"73694a08-761d-412e-9a30-5c02be8900dd\") "
Oct 30 20:17:29.355096 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:29.351637    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73694a08-761d-412e-9a30-5c02be8900dd-plugins-config-volume" pod "73694a08-761d-412e-9a30-5c02be8900dd" (UID: "73694a08-761d-412e-9a30-5c02be8900dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:29.355096 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:29.351997    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73694a08-761d-412e-9a30-5c02be8900dd-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:29.355096 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:29.353964    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73694a08-761d-412e-9a30-5c02be8900dd-kube-api-access-hmgr7" pod "73694a08-761d-412e-9a30-5c02be8900dd" (UID: "73694a08-761d-412e-9a30-5c02be8900dd"). InnerVolumeSpecName "kube-api-access-hmgr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:29.459705 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:29.459671    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmgr7\" (UniqueName: \"kubernetes.io/projected/73694a08-761d-412e-9a30-5c02be8900dd-kube-api-access-hmgr7\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:29.949148 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:29.949107    5958 generic.go:356] "Generic (PLEG): container finished" podID="73694a08-761d-412e-9a30-5c02be8900dd" containerID="0e208d30f988086800eea677204c765204521a97f9c60a5758271d3907d9b88b" exitCode=0
Oct 30 20:17:29.949444 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:29.949177    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482"
Oct 30 20:17:29.949652 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:29.949196    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482" event={"ID":"73694a08-761d-412e-9a30-5c02be8900dd","Type":"ContainerDied","Data":"0e208d30f988086800eea677204c765204521a97f9c60a5758271d3907d9b88b"}
Oct 30 20:17:29.949816 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:29.949800    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482" event={"ID":"73694a08-761d-412e-9a30-5c02be8900dd","Type":"ContainerDied","Data":"a130ea4fe56dd758753fc4040d3690ca18f11475bbee7f14750bed67b27235ba"}
Oct 30 20:17:29.949933 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:29.949920    5958 scope.go:122] "RemoveContainer" containerID="0e208d30f988086800eea677204c765204521a97f9c60a5758271d3907d9b88b"
Oct 30 20:17:29.978675 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:29.978633    5958 scope.go:122] "RemoveContainer" containerID="0e208d30f988086800eea677204c765204521a97f9c60a5758271d3907d9b88b"
Oct 30 20:17:29.980849 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:17:29.980816    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e208d30f988086800eea677204c765204521a97f9c60a5758271d3907d9b88b\": container with ID starting with 0e208d30f988086800eea677204c765204521a97f9c60a5758271d3907d9b88b not found: ID does not exist" containerID="0e208d30f988086800eea677204c765204521a97f9c60a5758271d3907d9b88b"
Oct 30 20:17:29.981036 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:29.981006    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e208d30f988086800eea677204c765204521a97f9c60a5758271d3907d9b88b"} err="failed to get container status \"0e208d30f988086800eea677204c765204521a97f9c60a5758271d3907d9b88b\": rpc error: code = NotFound desc = could not find container \"0e208d30f988086800eea677204c765204521a97f9c60a5758271d3907d9b88b\": container with ID starting with 0e208d30f988086800eea677204c765204521a97f9c60a5758271d3907d9b88b not found: ID does not exist"
Oct 30 20:17:30.117707 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:30.117659    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482"]
Oct 30 20:17:30.167218 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:30.167174    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3736ef9-dfbcbf74c-gr482"]
Oct 30 20:17:31.403082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:31.403043    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73694a08-761d-412e-9a30-5c02be8900dd" path="/var/lib/kubelet/pods/73694a08-761d-412e-9a30-5c02be8900dd/volumes"
Oct 30 20:17:36.611022 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:17:36.610976    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:17:41.779720 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:41.779684    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 20:17:41.783953 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:41.783928    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 20:17:51.589914 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:51.589875    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"]
Oct 30 20:17:51.590502 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:51.590111    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv" podUID="551cc517-12e5-49b1-beca-2ca8348fd2b8" containerName="istio-proxy" containerID="cri-o://5edc3281806b6a7be21e6ca69a21324c0ae73e6817a3b0e21bfd842e2964b45f" gracePeriod=30
Oct 30 20:17:51.640860 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:51.640822    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"]
Oct 30 20:17:51.641084 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:51.641060    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597" podUID="83b30dc9-50cf-4d71-9d59-b37e1e19f971" containerName="istio-proxy" containerID="cri-o://51114a8cf5ae3c66afea5a8d31b2050e87403eaa25266a7e3781e58a71c77cbb" gracePeriod=30
Oct 30 20:17:51.819027 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:51.818980    5958 patch_prober.go:29] interesting pod/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:17:51.819202 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:51.819044    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv" podUID="551cc517-12e5-49b1-beca-2ca8348fd2b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:17:52.196310 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:52.196268    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"]
Oct 30 20:17:52.196574 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:52.196550    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf" podUID="48c11b72-fdc6-4d2c-849b-136a5a53259d" containerName="istio-proxy" containerID="cri-o://57c38fb01c8ed968f02390fd03d537e541da06a81880e8dac502b98f9ba051cc" gracePeriod=30
Oct 30 20:17:52.477143 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:52.477047    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"]
Oct 30 20:17:52.477319 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:52.477291    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m" podUID="a977973c-341c-48ec-9df5-8035da148b90" containerName="istio-proxy" containerID="cri-o://956d556026f44b19e237d54461eea3f29e9376fab24aa4aad16d5348f55964ff" gracePeriod=30
Oct 30 20:17:52.481032 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:52.481003    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"]
Oct 30 20:17:52.481214 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:52.481194    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm" podUID="3017938e-ba45-4d82-b5bb-15919e2b413e" containerName="istio-proxy" containerID="cri-o://9dde582e812e89b47ecf1f64de36c322351fc581e5cca5855f9992b25b87c446" gracePeriod=30
Oct 30 20:17:52.953542 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:52.953496    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"]
Oct 30 20:17:52.953993 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:52.953714    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq" podUID="0a7169d8-4a5a-486d-840c-20a92d74bef5" containerName="istio-proxy" containerID="cri-o://12b1e92d1e057f26118e0185e8988067370b9a8c83fa962073a8d7677cd5aef7" gracePeriod=30
Oct 30 20:17:53.256593 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:53.256495    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"]
Oct 30 20:17:53.256758 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:53.256727    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89" podUID="cc641430-3d40-4d0d-b2c6-b551b090da1f" containerName="istio-proxy" containerID="cri-o://94028ad1b4ccceb187ceb8c0143eece317dd6904d953ba6bcb74b45d5a6cfcb3" gracePeriod=30
Oct 30 20:17:54.891412 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:54.891374    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:17:54.891838 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:54.891490    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq" podUID="0a7169d8-4a5a-486d-840c-20a92d74bef5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:17:55.922233 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:55.922190    5958 patch_prober.go:29] interesting pod/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:17:55.922745 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:55.922261    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89" podUID="cc641430-3d40-4d0d-b2c6-b551b090da1f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:17:56.016353 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:56.016313    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89"]
Oct 30 20:17:56.093779 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:56.093732    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:17:56.093946 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:56.093794    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m" podUID="a977973c-341c-48ec-9df5-8035da148b90" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:17:56.479105 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:56.479048    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"]
Oct 30 20:17:56.479684 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:56.479649    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4" podUID="3b3d965b-503e-44f3-b95d-b7c599310b81" containerName="istio-proxy" containerID="cri-o://b2e55881e5f45d47e7da5fb443c1edc0aa0abc8a738898cb087e57364b63506e" gracePeriod=30
Oct 30 20:17:56.517492 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:56.514878    5958 patch_prober.go:29] interesting pod/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:17:56.517492 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:56.514981    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm" podUID="3017938e-ba45-4d82-b5bb-15919e2b413e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:17:56.946794 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:56.946767    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:17:57.043138 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.043034    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b30dc9-50cf-4d71-9d59-b37e1e19f971-kube-api-access-42ccg\" (UniqueName: \"kubernetes.io/projected/83b30dc9-50cf-4d71-9d59-b37e1e19f971-kube-api-access-42ccg\") pod \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") "
Oct 30 20:17:57.043332 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.043196    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-workload-certs\") pod \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") "
Oct 30 20:17:57.043332 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.043285    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-podinfo\") pod \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") "
Oct 30 20:17:57.043516 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.043410    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-token\" (UniqueName: \"kubernetes.io/projected/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-token\") pod \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") "
Oct 30 20:17:57.043588 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.043540    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-data\") pod \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") "
Oct 30 20:17:57.043642 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.043576    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-workload-certs" pod "83b30dc9-50cf-4d71-9d59-b37e1e19f971" (UID: "83b30dc9-50cf-4d71-9d59-b37e1e19f971"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:57.043642 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.043618    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-workload-socket\") pod \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") "
Oct 30 20:17:57.043748 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.043688    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istiod-ca-cert\") pod \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") "
Oct 30 20:17:57.043801 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.043766    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-envoy\") pod \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") "
Oct 30 20:17:57.043887 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.043855    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-credential-socket\") pod \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\" (UID: \"83b30dc9-50cf-4d71-9d59-b37e1e19f971\") "
Oct 30 20:17:57.043887 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.043861    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-data" pod "83b30dc9-50cf-4d71-9d59-b37e1e19f971" (UID: "83b30dc9-50cf-4d71-9d59-b37e1e19f971"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:57.043997 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.043963    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-workload-socket" pod "83b30dc9-50cf-4d71-9d59-b37e1e19f971" (UID: "83b30dc9-50cf-4d71-9d59-b37e1e19f971"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:57.044174 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.044103    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istiod-ca-cert" pod "83b30dc9-50cf-4d71-9d59-b37e1e19f971" (UID: "83b30dc9-50cf-4d71-9d59-b37e1e19f971"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:57.044324 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.044304    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-credential-socket" pod "83b30dc9-50cf-4d71-9d59-b37e1e19f971" (UID: "83b30dc9-50cf-4d71-9d59-b37e1e19f971"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:57.045003 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.044974    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.045110 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.045007    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.045110 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.045025    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.045110 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.045041    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.045110 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.045055    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.046423 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.046397    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-token" pod "83b30dc9-50cf-4d71-9d59-b37e1e19f971" (UID: "83b30dc9-50cf-4d71-9d59-b37e1e19f971"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:57.046555 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.046425    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-podinfo" pod "83b30dc9-50cf-4d71-9d59-b37e1e19f971" (UID: "83b30dc9-50cf-4d71-9d59-b37e1e19f971"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:17:57.046555 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.046427    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b30dc9-50cf-4d71-9d59-b37e1e19f971-kube-api-access-42ccg" pod "83b30dc9-50cf-4d71-9d59-b37e1e19f971" (UID: "83b30dc9-50cf-4d71-9d59-b37e1e19f971"). InnerVolumeSpecName "kube-api-access-42ccg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:57.046671 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.046653    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-envoy" pod "83b30dc9-50cf-4d71-9d59-b37e1e19f971" (UID: "83b30dc9-50cf-4d71-9d59-b37e1e19f971"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:57.146374 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.146331    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42ccg\" (UniqueName: \"kubernetes.io/projected/83b30dc9-50cf-4d71-9d59-b37e1e19f971-kube-api-access-42ccg\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.146374 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.146366    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.146374 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.146379    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.146618 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.146390    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b30dc9-50cf-4d71-9d59-b37e1e19f971-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.229531 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.229495    5958 generic.go:356] "Generic (PLEG): container finished" podID="551cc517-12e5-49b1-beca-2ca8348fd2b8" containerID="5edc3281806b6a7be21e6ca69a21324c0ae73e6817a3b0e21bfd842e2964b45f" exitCode=0
Oct 30 20:17:57.229707 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.229576    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv" event={"ID":"551cc517-12e5-49b1-beca-2ca8348fd2b8","Type":"ContainerDied","Data":"5edc3281806b6a7be21e6ca69a21324c0ae73e6817a3b0e21bfd842e2964b45f"}
Oct 30 20:17:57.231550 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.231518    5958 generic.go:356] "Generic (PLEG): container finished" podID="83b30dc9-50cf-4d71-9d59-b37e1e19f971" containerID="51114a8cf5ae3c66afea5a8d31b2050e87403eaa25266a7e3781e58a71c77cbb" exitCode=0
Oct 30 20:17:57.231679 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.231566    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597" event={"ID":"83b30dc9-50cf-4d71-9d59-b37e1e19f971","Type":"ContainerDied","Data":"51114a8cf5ae3c66afea5a8d31b2050e87403eaa25266a7e3781e58a71c77cbb"}
Oct 30 20:17:57.231679 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.231580    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"
Oct 30 20:17:57.231679 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.231619    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597" event={"ID":"83b30dc9-50cf-4d71-9d59-b37e1e19f971","Type":"ContainerDied","Data":"6f8f3f8f437e1eb3fb717fe18455d523a313d860d4a37d7824bae687579ba04f"}
Oct 30 20:17:57.231679 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.231651    5958 scope.go:122] "RemoveContainer" containerID="51114a8cf5ae3c66afea5a8d31b2050e87403eaa25266a7e3781e58a71c77cbb"
Oct 30 20:17:57.313001 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.312981    5958 scope.go:122] "RemoveContainer" containerID="51114a8cf5ae3c66afea5a8d31b2050e87403eaa25266a7e3781e58a71c77cbb"
Oct 30 20:17:57.313391 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:17:57.313359    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51114a8cf5ae3c66afea5a8d31b2050e87403eaa25266a7e3781e58a71c77cbb\": container with ID starting with 51114a8cf5ae3c66afea5a8d31b2050e87403eaa25266a7e3781e58a71c77cbb not found: ID does not exist" containerID="51114a8cf5ae3c66afea5a8d31b2050e87403eaa25266a7e3781e58a71c77cbb"
Oct 30 20:17:57.313584 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.313403    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51114a8cf5ae3c66afea5a8d31b2050e87403eaa25266a7e3781e58a71c77cbb"} err="failed to get container status \"51114a8cf5ae3c66afea5a8d31b2050e87403eaa25266a7e3781e58a71c77cbb\": rpc error: code = NotFound desc = could not find container \"51114a8cf5ae3c66afea5a8d31b2050e87403eaa25266a7e3781e58a71c77cbb\": container with ID starting with 51114a8cf5ae3c66afea5a8d31b2050e87403eaa25266a7e3781e58a71c77cbb not found: ID does not exist"
Oct 30 20:17:57.320182 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.320037    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:17:57.428400 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.428373    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"]
Oct 30 20:17:57.428583 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.428415    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-wv597"]
Oct 30 20:17:57.449997 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.449951    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551cc517-12e5-49b1-beca-2ca8348fd2b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551cc517-12e5-49b1-beca-2ca8348fd2b8-istiod-ca-cert\") pod \"551cc517-12e5-49b1-beca-2ca8348fd2b8\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") "
Oct 30 20:17:57.450196 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.450014    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-workload-socket\") pod \"551cc517-12e5-49b1-beca-2ca8348fd2b8\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") "
Oct 30 20:17:57.450196 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.450109    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-envoy\") pod \"551cc517-12e5-49b1-beca-2ca8348fd2b8\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") "
Oct 30 20:17:57.450196 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.450192    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-podinfo\") pod \"551cc517-12e5-49b1-beca-2ca8348fd2b8\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") "
Oct 30 20:17:57.450440 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.450273    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-workload-certs\") pod \"551cc517-12e5-49b1-beca-2ca8348fd2b8\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") "
Oct 30 20:17:57.450440 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.450388    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-credential-socket\") pod \"551cc517-12e5-49b1-beca-2ca8348fd2b8\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") "
Oct 30 20:17:57.450440 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.450428    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551cc517-12e5-49b1-beca-2ca8348fd2b8-kube-api-access-65xq4\" (UniqueName: \"kubernetes.io/projected/551cc517-12e5-49b1-beca-2ca8348fd2b8-kube-api-access-65xq4\") pod \"551cc517-12e5-49b1-beca-2ca8348fd2b8\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") "
Oct 30 20:17:57.450679 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.450478    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551cc517-12e5-49b1-beca-2ca8348fd2b8-istiod-ca-cert" pod "551cc517-12e5-49b1-beca-2ca8348fd2b8" (UID: "551cc517-12e5-49b1-beca-2ca8348fd2b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:57.450679 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.450588    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-data\") pod \"551cc517-12e5-49b1-beca-2ca8348fd2b8\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") "
Oct 30 20:17:57.450679 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.450614    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-workload-certs" pod "551cc517-12e5-49b1-beca-2ca8348fd2b8" (UID: "551cc517-12e5-49b1-beca-2ca8348fd2b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:57.450864 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.450679    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-token\" (UniqueName: \"kubernetes.io/projected/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-token\") pod \"551cc517-12e5-49b1-beca-2ca8348fd2b8\" (UID: \"551cc517-12e5-49b1-beca-2ca8348fd2b8\") "
Oct 30 20:17:57.450864 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.450738    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-credential-socket" pod "551cc517-12e5-49b1-beca-2ca8348fd2b8" (UID: "551cc517-12e5-49b1-beca-2ca8348fd2b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:57.450978 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.450915    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-data" pod "551cc517-12e5-49b1-beca-2ca8348fd2b8" (UID: "551cc517-12e5-49b1-beca-2ca8348fd2b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:57.451348 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.451317    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-workload-socket" pod "551cc517-12e5-49b1-beca-2ca8348fd2b8" (UID: "551cc517-12e5-49b1-beca-2ca8348fd2b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:57.451530 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.451511    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.451615 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.451535    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.451615 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.451551    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551cc517-12e5-49b1-beca-2ca8348fd2b8-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.451615 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.451564    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.451615 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.451576    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.453808 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.453765    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-token" pod "551cc517-12e5-49b1-beca-2ca8348fd2b8" (UID: "551cc517-12e5-49b1-beca-2ca8348fd2b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:57.454291 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.454268    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551cc517-12e5-49b1-beca-2ca8348fd2b8-kube-api-access-65xq4" pod "551cc517-12e5-49b1-beca-2ca8348fd2b8" (UID: "551cc517-12e5-49b1-beca-2ca8348fd2b8"). InnerVolumeSpecName "kube-api-access-65xq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:57.454387 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.454342    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-envoy" pod "551cc517-12e5-49b1-beca-2ca8348fd2b8" (UID: "551cc517-12e5-49b1-beca-2ca8348fd2b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:57.454532 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.454507    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-podinfo" pod "551cc517-12e5-49b1-beca-2ca8348fd2b8" (UID: "551cc517-12e5-49b1-beca-2ca8348fd2b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:17:57.475547 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.475501    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:17:57.553419 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.553382    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.553419 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.553416    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.553419 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.553427    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551cc517-12e5-49b1-beca-2ca8348fd2b8-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.553698 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.553437    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65xq4\" (UniqueName: \"kubernetes.io/projected/551cc517-12e5-49b1-beca-2ca8348fd2b8-kube-api-access-65xq4\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.655517 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.655438    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c11b72-fdc6-4d2c-849b-136a5a53259d-kube-api-access-kf6z6\" (UniqueName: \"kubernetes.io/projected/48c11b72-fdc6-4d2c-849b-136a5a53259d-kube-api-access-kf6z6\") pod \"48c11b72-fdc6-4d2c-849b-136a5a53259d\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") "
Oct 30 20:17:57.657835 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.657797    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-workload-certs\") pod \"48c11b72-fdc6-4d2c-849b-136a5a53259d\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") "
Oct 30 20:17:57.658023 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.657910    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-credential-socket\") pod \"48c11b72-fdc6-4d2c-849b-136a5a53259d\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") "
Oct 30 20:17:57.658023 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.657946    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-podinfo\") pod \"48c11b72-fdc6-4d2c-849b-136a5a53259d\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") "
Oct 30 20:17:57.658136 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.658020    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48c11b72-fdc6-4d2c-849b-136a5a53259d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c11b72-fdc6-4d2c-849b-136a5a53259d-istiod-ca-cert\") pod \"48c11b72-fdc6-4d2c-849b-136a5a53259d\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") "
Oct 30 20:17:57.658136 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.658083    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-data\") pod \"48c11b72-fdc6-4d2c-849b-136a5a53259d\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") "
Oct 30 20:17:57.658241 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.658224    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-workload-socket\") pod \"48c11b72-fdc6-4d2c-849b-136a5a53259d\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") "
Oct 30 20:17:57.658315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.658299    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-envoy\") pod \"48c11b72-fdc6-4d2c-849b-136a5a53259d\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") "
Oct 30 20:17:57.658372 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.658344    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-credential-socket" pod "48c11b72-fdc6-4d2c-849b-136a5a53259d" (UID: "48c11b72-fdc6-4d2c-849b-136a5a53259d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:57.658427 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.658371    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-token\" (UniqueName: \"kubernetes.io/projected/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-token\") pod \"48c11b72-fdc6-4d2c-849b-136a5a53259d\" (UID: \"48c11b72-fdc6-4d2c-849b-136a5a53259d\") "
Oct 30 20:17:57.658586 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.658554    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-workload-certs" pod "48c11b72-fdc6-4d2c-849b-136a5a53259d" (UID: "48c11b72-fdc6-4d2c-849b-136a5a53259d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:57.658760 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.658742    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-data" pod "48c11b72-fdc6-4d2c-849b-136a5a53259d" (UID: "48c11b72-fdc6-4d2c-849b-136a5a53259d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:57.659290 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.659224    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.659290 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.659267    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.659290 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.659283    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.659660 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.659592    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-workload-socket" pod "48c11b72-fdc6-4d2c-849b-136a5a53259d" (UID: "48c11b72-fdc6-4d2c-849b-136a5a53259d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:57.659741 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.659682    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c11b72-fdc6-4d2c-849b-136a5a53259d-istiod-ca-cert" pod "48c11b72-fdc6-4d2c-849b-136a5a53259d" (UID: "48c11b72-fdc6-4d2c-849b-136a5a53259d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:57.667956 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.661522    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-token" pod "48c11b72-fdc6-4d2c-849b-136a5a53259d" (UID: "48c11b72-fdc6-4d2c-849b-136a5a53259d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:57.667956 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.662170    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-envoy" pod "48c11b72-fdc6-4d2c-849b-136a5a53259d" (UID: "48c11b72-fdc6-4d2c-849b-136a5a53259d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:57.673962 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.673905    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c11b72-fdc6-4d2c-849b-136a5a53259d-kube-api-access-kf6z6" pod "48c11b72-fdc6-4d2c-849b-136a5a53259d" (UID: "48c11b72-fdc6-4d2c-849b-136a5a53259d"). InnerVolumeSpecName "kube-api-access-kf6z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:57.674620 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.674560    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-podinfo" pod "48c11b72-fdc6-4d2c-849b-136a5a53259d" (UID: "48c11b72-fdc6-4d2c-849b-136a5a53259d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:17:57.766946 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.760972    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.766946 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.761013    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf6z6\" (UniqueName: \"kubernetes.io/projected/48c11b72-fdc6-4d2c-849b-136a5a53259d-kube-api-access-kf6z6\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.766946 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.761029    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.766946 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.761043    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c11b72-fdc6-4d2c-849b-136a5a53259d-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.766946 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.761057    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:57.766946 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.761070    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c11b72-fdc6-4d2c-849b-136a5a53259d-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.003699 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:57.999441    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:17:58.017219 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.017182    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:17:58.054940 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.054901    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-credential-socket\") pod \"a977973c-341c-48ec-9df5-8035da148b90\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") "
Oct 30 20:17:58.055119 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.054971    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-workload-certs\") pod \"a977973c-341c-48ec-9df5-8035da148b90\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") "
Oct 30 20:17:58.055119 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.055053    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a977973c-341c-48ec-9df5-8035da148b90-istio-token\" (UniqueName: \"kubernetes.io/projected/a977973c-341c-48ec-9df5-8035da148b90-istio-token\") pod \"a977973c-341c-48ec-9df5-8035da148b90\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") "
Oct 30 20:17:58.055119 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.055082    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a977973c-341c-48ec-9df5-8035da148b90-kube-api-access-5xjb4\" (UniqueName: \"kubernetes.io/projected/a977973c-341c-48ec-9df5-8035da148b90-kube-api-access-5xjb4\") pod \"a977973c-341c-48ec-9df5-8035da148b90\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") "
Oct 30 20:17:58.055119 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.055110    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a977973c-341c-48ec-9df5-8035da148b90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a977973c-341c-48ec-9df5-8035da148b90-istio-podinfo\") pod \"a977973c-341c-48ec-9df5-8035da148b90\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") "
Oct 30 20:17:58.055360 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.055220    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-istio-data\") pod \"a977973c-341c-48ec-9df5-8035da148b90\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") "
Oct 30 20:17:58.055360 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.055271    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-credential-socket" pod "a977973c-341c-48ec-9df5-8035da148b90" (UID: "a977973c-341c-48ec-9df5-8035da148b90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.055360 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.055288    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-workload-certs" pod "a977973c-341c-48ec-9df5-8035da148b90" (UID: "a977973c-341c-48ec-9df5-8035da148b90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.055538 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.055366    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-istio-envoy\") pod \"a977973c-341c-48ec-9df5-8035da148b90\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") "
Oct 30 20:17:58.055538 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.055410    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a977973c-341c-48ec-9df5-8035da148b90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a977973c-341c-48ec-9df5-8035da148b90-istiod-ca-cert\") pod \"a977973c-341c-48ec-9df5-8035da148b90\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") "
Oct 30 20:17:58.055538 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.055508    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-istio-data" pod "a977973c-341c-48ec-9df5-8035da148b90" (UID: "a977973c-341c-48ec-9df5-8035da148b90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.055538 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.055521    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-workload-socket\") pod \"a977973c-341c-48ec-9df5-8035da148b90\" (UID: \"a977973c-341c-48ec-9df5-8035da148b90\") "
Oct 30 20:17:58.056634 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.056600    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-workload-socket" pod "a977973c-341c-48ec-9df5-8035da148b90" (UID: "a977973c-341c-48ec-9df5-8035da148b90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.056850 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.056606    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.056919 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.056868    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.057004 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.056990    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.057065 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.057018    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a977973c-341c-48ec-9df5-8035da148b90-istiod-ca-cert" pod "a977973c-341c-48ec-9df5-8035da148b90" (UID: "a977973c-341c-48ec-9df5-8035da148b90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:58.058257 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.058226    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a977973c-341c-48ec-9df5-8035da148b90-istio-podinfo" pod "a977973c-341c-48ec-9df5-8035da148b90" (UID: "a977973c-341c-48ec-9df5-8035da148b90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:17:58.058896 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.058864    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a977973c-341c-48ec-9df5-8035da148b90-kube-api-access-5xjb4" pod "a977973c-341c-48ec-9df5-8035da148b90" (UID: "a977973c-341c-48ec-9df5-8035da148b90"). InnerVolumeSpecName "kube-api-access-5xjb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:58.060271 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.060249    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a977973c-341c-48ec-9df5-8035da148b90-istio-token" pod "a977973c-341c-48ec-9df5-8035da148b90" (UID: "a977973c-341c-48ec-9df5-8035da148b90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:58.060534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.060516    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-istio-envoy" pod "a977973c-341c-48ec-9df5-8035da148b90" (UID: "a977973c-341c-48ec-9df5-8035da148b90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.158603 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.157669    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-workload-socket\") pod \"3017938e-ba45-4d82-b5bb-15919e2b413e\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") "
Oct 30 20:17:58.158603 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.157753    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3017938e-ba45-4d82-b5bb-15919e2b413e-kube-api-access-xdqfz\" (UniqueName: \"kubernetes.io/projected/3017938e-ba45-4d82-b5bb-15919e2b413e-kube-api-access-xdqfz\") pod \"3017938e-ba45-4d82-b5bb-15919e2b413e\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") "
Oct 30 20:17:58.158603 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.157981    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-credential-socket\") pod \"3017938e-ba45-4d82-b5bb-15919e2b413e\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") "
Oct 30 20:17:58.158603 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.158072    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-workload-certs\") pod \"3017938e-ba45-4d82-b5bb-15919e2b413e\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") "
Oct 30 20:17:58.158603 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.158147    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-data\") pod \"3017938e-ba45-4d82-b5bb-15919e2b413e\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") "
Oct 30 20:17:58.158603 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.157977    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-workload-socket" pod "3017938e-ba45-4d82-b5bb-15919e2b413e" (UID: "3017938e-ba45-4d82-b5bb-15919e2b413e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.158603 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.158257    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-envoy\") pod \"3017938e-ba45-4d82-b5bb-15919e2b413e\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") "
Oct 30 20:17:58.158603 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.158302    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-token\" (UniqueName: \"kubernetes.io/projected/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-token\") pod \"3017938e-ba45-4d82-b5bb-15919e2b413e\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") "
Oct 30 20:17:58.158603 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.158352    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3017938e-ba45-4d82-b5bb-15919e2b413e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3017938e-ba45-4d82-b5bb-15919e2b413e-istiod-ca-cert\") pod \"3017938e-ba45-4d82-b5bb-15919e2b413e\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") "
Oct 30 20:17:58.158603 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.158386    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-podinfo\") pod \"3017938e-ba45-4d82-b5bb-15919e2b413e\" (UID: \"3017938e-ba45-4d82-b5bb-15919e2b413e\") "
Oct 30 20:17:58.159273 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.158636    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-credential-socket" pod "3017938e-ba45-4d82-b5bb-15919e2b413e" (UID: "3017938e-ba45-4d82-b5bb-15919e2b413e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.159273 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.159161    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.159273 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.159189    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a977973c-341c-48ec-9df5-8035da148b90-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.159273 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.159204    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xjb4\" (UniqueName: \"kubernetes.io/projected/a977973c-341c-48ec-9df5-8035da148b90-kube-api-access-5xjb4\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.159273 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.159220    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a977973c-341c-48ec-9df5-8035da148b90-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.159273 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.159234    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.159273 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.159248    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.159273 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.159262    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a977973c-341c-48ec-9df5-8035da148b90-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.159273 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.159274    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a977973c-341c-48ec-9df5-8035da148b90-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.160172 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.160103    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-data" pod "3017938e-ba45-4d82-b5bb-15919e2b413e" (UID: "3017938e-ba45-4d82-b5bb-15919e2b413e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.160277 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.160194    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-workload-certs" pod "3017938e-ba45-4d82-b5bb-15919e2b413e" (UID: "3017938e-ba45-4d82-b5bb-15919e2b413e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.160820 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.160697    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3017938e-ba45-4d82-b5bb-15919e2b413e-istiod-ca-cert" pod "3017938e-ba45-4d82-b5bb-15919e2b413e" (UID: "3017938e-ba45-4d82-b5bb-15919e2b413e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:58.161599 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.161570    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3017938e-ba45-4d82-b5bb-15919e2b413e-kube-api-access-xdqfz" pod "3017938e-ba45-4d82-b5bb-15919e2b413e" (UID: "3017938e-ba45-4d82-b5bb-15919e2b413e"). InnerVolumeSpecName "kube-api-access-xdqfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:58.163865 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.163835    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-envoy" pod "3017938e-ba45-4d82-b5bb-15919e2b413e" (UID: "3017938e-ba45-4d82-b5bb-15919e2b413e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.165389 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.165361    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-podinfo" pod "3017938e-ba45-4d82-b5bb-15919e2b413e" (UID: "3017938e-ba45-4d82-b5bb-15919e2b413e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:17:58.166331 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.166304    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-token" pod "3017938e-ba45-4d82-b5bb-15919e2b413e" (UID: "3017938e-ba45-4d82-b5bb-15919e2b413e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:58.229830 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.229797    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:17:58.256820 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.256782    5958 generic.go:356] "Generic (PLEG): container finished" podID="48c11b72-fdc6-4d2c-849b-136a5a53259d" containerID="57c38fb01c8ed968f02390fd03d537e541da06a81880e8dac502b98f9ba051cc" exitCode=0
Oct 30 20:17:58.256996 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.256841    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"
Oct 30 20:17:58.256996 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.256876    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf" event={"ID":"48c11b72-fdc6-4d2c-849b-136a5a53259d","Type":"ContainerDied","Data":"57c38fb01c8ed968f02390fd03d537e541da06a81880e8dac502b98f9ba051cc"}
Oct 30 20:17:58.256996 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.256955    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf" event={"ID":"48c11b72-fdc6-4d2c-849b-136a5a53259d","Type":"ContainerDied","Data":"77ed7422a2e5e13dfeb3e1a582d268995f47705a00c2330470e7d6d714485f2c"}
Oct 30 20:17:58.256996 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.256988    5958 scope.go:122] "RemoveContainer" containerID="57c38fb01c8ed968f02390fd03d537e541da06a81880e8dac502b98f9ba051cc"
Oct 30 20:17:58.260756 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.260723    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdqfz\" (UniqueName: \"kubernetes.io/projected/3017938e-ba45-4d82-b5bb-15919e2b413e-kube-api-access-xdqfz\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.260756 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.260759    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.260999 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.260776    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.260999 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.260790    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.260999 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.260805    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.260999 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.260821    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3017938e-ba45-4d82-b5bb-15919e2b413e-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.260999 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.260837    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3017938e-ba45-4d82-b5bb-15919e2b413e-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.270695 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.265277    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"
Oct 30 20:17:58.270695 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.265300    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv" event={"ID":"551cc517-12e5-49b1-beca-2ca8348fd2b8","Type":"ContainerDied","Data":"6521a6eeb75f86faf9e556be8f4a94624c54de2d5e696a5218b1e4328b57d039"}
Oct 30 20:17:58.278535 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.274936    5958 generic.go:356] "Generic (PLEG): container finished" podID="3017938e-ba45-4d82-b5bb-15919e2b413e" containerID="9dde582e812e89b47ecf1f64de36c322351fc581e5cca5855f9992b25b87c446" exitCode=0
Oct 30 20:17:58.278535 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.275074    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm" event={"ID":"3017938e-ba45-4d82-b5bb-15919e2b413e","Type":"ContainerDied","Data":"9dde582e812e89b47ecf1f64de36c322351fc581e5cca5855f9992b25b87c446"}
Oct 30 20:17:58.278535 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.275127    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm" event={"ID":"3017938e-ba45-4d82-b5bb-15919e2b413e","Type":"ContainerDied","Data":"7fef5f007078a6f6c6a9478fd71c8ac293e1f42d7d4942a2be8644227227de6e"}
Oct 30 20:17:58.278535 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.275216    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"
Oct 30 20:17:58.281665 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.280672    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"
Oct 30 20:17:58.282193 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.282111    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m" event={"ID":"a977973c-341c-48ec-9df5-8035da148b90","Type":"ContainerDied","Data":"956d556026f44b19e237d54461eea3f29e9376fab24aa4aad16d5348f55964ff"}
Oct 30 20:17:58.284185 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.282739    5958 generic.go:356] "Generic (PLEG): container finished" podID="a977973c-341c-48ec-9df5-8035da148b90" containerID="956d556026f44b19e237d54461eea3f29e9376fab24aa4aad16d5348f55964ff" exitCode=0
Oct 30 20:17:58.284185 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.282859    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m" event={"ID":"a977973c-341c-48ec-9df5-8035da148b90","Type":"ContainerDied","Data":"0c6bff43f51631384279a5eeade6815013ad2ba24e4b67e0cffda5b2c41d0273"}
Oct 30 20:17:58.285034 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.284763    5958 generic.go:356] "Generic (PLEG): container finished" podID="0a7169d8-4a5a-486d-840c-20a92d74bef5" containerID="12b1e92d1e057f26118e0185e8988067370b9a8c83fa962073a8d7677cd5aef7" exitCode=0
Oct 30 20:17:58.285034 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.284819    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"
Oct 30 20:17:58.285034 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.284834    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq" event={"ID":"0a7169d8-4a5a-486d-840c-20a92d74bef5","Type":"ContainerDied","Data":"12b1e92d1e057f26118e0185e8988067370b9a8c83fa962073a8d7677cd5aef7"}
Oct 30 20:17:58.285034 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.284873    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq" event={"ID":"0a7169d8-4a5a-486d-840c-20a92d74bef5","Type":"ContainerDied","Data":"c8324b8a6b79a3d3f117724cbfc9ea98409ce46e5ed6533ca5d1fbf1249f0288"}
Oct 30 20:17:58.294916 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.294759    5958 scope.go:122] "RemoveContainer" containerID="57c38fb01c8ed968f02390fd03d537e541da06a81880e8dac502b98f9ba051cc"
Oct 30 20:17:58.295113 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:17:58.295082    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57c38fb01c8ed968f02390fd03d537e541da06a81880e8dac502b98f9ba051cc\": container with ID starting with 57c38fb01c8ed968f02390fd03d537e541da06a81880e8dac502b98f9ba051cc not found: ID does not exist" containerID="57c38fb01c8ed968f02390fd03d537e541da06a81880e8dac502b98f9ba051cc"
Oct 30 20:17:58.295187 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.295127    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57c38fb01c8ed968f02390fd03d537e541da06a81880e8dac502b98f9ba051cc"} err="failed to get container status \"57c38fb01c8ed968f02390fd03d537e541da06a81880e8dac502b98f9ba051cc\": rpc error: code = NotFound desc = could not find container \"57c38fb01c8ed968f02390fd03d537e541da06a81880e8dac502b98f9ba051cc\": container with ID starting with 57c38fb01c8ed968f02390fd03d537e541da06a81880e8dac502b98f9ba051cc not found: ID does not exist"
Oct 30 20:17:58.295187 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.295152    5958 scope.go:122] "RemoveContainer" containerID="5edc3281806b6a7be21e6ca69a21324c0ae73e6817a3b0e21bfd842e2964b45f"
Oct 30 20:17:58.312429 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.312396    5958 scope.go:122] "RemoveContainer" containerID="9dde582e812e89b47ecf1f64de36c322351fc581e5cca5855f9992b25b87c446"
Oct 30 20:17:58.324919 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.324897    5958 scope.go:122] "RemoveContainer" containerID="9dde582e812e89b47ecf1f64de36c322351fc581e5cca5855f9992b25b87c446"
Oct 30 20:17:58.325298 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:17:58.325274    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dde582e812e89b47ecf1f64de36c322351fc581e5cca5855f9992b25b87c446\": container with ID starting with 9dde582e812e89b47ecf1f64de36c322351fc581e5cca5855f9992b25b87c446 not found: ID does not exist" containerID="9dde582e812e89b47ecf1f64de36c322351fc581e5cca5855f9992b25b87c446"
Oct 30 20:17:58.325356 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.325307    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dde582e812e89b47ecf1f64de36c322351fc581e5cca5855f9992b25b87c446"} err="failed to get container status \"9dde582e812e89b47ecf1f64de36c322351fc581e5cca5855f9992b25b87c446\": rpc error: code = NotFound desc = could not find container \"9dde582e812e89b47ecf1f64de36c322351fc581e5cca5855f9992b25b87c446\": container with ID starting with 9dde582e812e89b47ecf1f64de36c322351fc581e5cca5855f9992b25b87c446 not found: ID does not exist"
Oct 30 20:17:58.325356 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.325328    5958 scope.go:122] "RemoveContainer" containerID="956d556026f44b19e237d54461eea3f29e9376fab24aa4aad16d5348f55964ff"
Oct 30 20:17:58.362486 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.362083    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a7169d8-4a5a-486d-840c-20a92d74bef5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a7169d8-4a5a-486d-840c-20a92d74bef5-istiod-ca-cert\") pod \"0a7169d8-4a5a-486d-840c-20a92d74bef5\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") "
Oct 30 20:17:58.362486 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.362401    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-workload-certs\") pod \"0a7169d8-4a5a-486d-840c-20a92d74bef5\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") "
Oct 30 20:17:58.362742 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.362496    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a7169d8-4a5a-486d-840c-20a92d74bef5-istiod-ca-cert" pod "0a7169d8-4a5a-486d-840c-20a92d74bef5" (UID: "0a7169d8-4a5a-486d-840c-20a92d74bef5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:58.362929 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.362843    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-data\") pod \"0a7169d8-4a5a-486d-840c-20a92d74bef5\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") "
Oct 30 20:17:58.364177 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.363083    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-workload-socket\") pod \"0a7169d8-4a5a-486d-840c-20a92d74bef5\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") "
Oct 30 20:17:58.364177 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.363138    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-podinfo\") pod \"0a7169d8-4a5a-486d-840c-20a92d74bef5\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") "
Oct 30 20:17:58.364177 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.363186    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-credential-socket\") pod \"0a7169d8-4a5a-486d-840c-20a92d74bef5\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") "
Oct 30 20:17:58.364177 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.363239    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-envoy\") pod \"0a7169d8-4a5a-486d-840c-20a92d74bef5\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") "
Oct 30 20:17:58.364177 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.363272    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a7169d8-4a5a-486d-840c-20a92d74bef5-kube-api-access-z7csq\" (UniqueName: \"kubernetes.io/projected/0a7169d8-4a5a-486d-840c-20a92d74bef5-kube-api-access-z7csq\") pod \"0a7169d8-4a5a-486d-840c-20a92d74bef5\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") "
Oct 30 20:17:58.364177 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.363349    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-token\" (UniqueName: \"kubernetes.io/projected/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-token\") pod \"0a7169d8-4a5a-486d-840c-20a92d74bef5\" (UID: \"0a7169d8-4a5a-486d-840c-20a92d74bef5\") "
Oct 30 20:17:58.364177 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.364106    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a7169d8-4a5a-486d-840c-20a92d74bef5-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.367437 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.367381    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-workload-certs" pod "0a7169d8-4a5a-486d-840c-20a92d74bef5" (UID: "0a7169d8-4a5a-486d-840c-20a92d74bef5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.369667 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.369544    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-workload-socket" pod "0a7169d8-4a5a-486d-840c-20a92d74bef5" (UID: "0a7169d8-4a5a-486d-840c-20a92d74bef5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.369818 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.369683    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-credential-socket" pod "0a7169d8-4a5a-486d-840c-20a92d74bef5" (UID: "0a7169d8-4a5a-486d-840c-20a92d74bef5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.401700 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.377682    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a7169d8-4a5a-486d-840c-20a92d74bef5-kube-api-access-z7csq" pod "0a7169d8-4a5a-486d-840c-20a92d74bef5" (UID: "0a7169d8-4a5a-486d-840c-20a92d74bef5"). InnerVolumeSpecName "kube-api-access-z7csq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:58.401700 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.378162    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-podinfo" pod "0a7169d8-4a5a-486d-840c-20a92d74bef5" (UID: "0a7169d8-4a5a-486d-840c-20a92d74bef5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:17:58.401700 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.378327    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-token" pod "0a7169d8-4a5a-486d-840c-20a92d74bef5" (UID: "0a7169d8-4a5a-486d-840c-20a92d74bef5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:58.401700 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.378989    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-envoy" pod "0a7169d8-4a5a-486d-840c-20a92d74bef5" (UID: "0a7169d8-4a5a-486d-840c-20a92d74bef5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.401700 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.379656    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-data" pod "0a7169d8-4a5a-486d-840c-20a92d74bef5" (UID: "0a7169d8-4a5a-486d-840c-20a92d74bef5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.401700 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.399070    5958 scope.go:122] "RemoveContainer" containerID="956d556026f44b19e237d54461eea3f29e9376fab24aa4aad16d5348f55964ff"
Oct 30 20:17:58.401700 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:17:58.399482    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"956d556026f44b19e237d54461eea3f29e9376fab24aa4aad16d5348f55964ff\": container with ID starting with 956d556026f44b19e237d54461eea3f29e9376fab24aa4aad16d5348f55964ff not found: ID does not exist" containerID="956d556026f44b19e237d54461eea3f29e9376fab24aa4aad16d5348f55964ff"
Oct 30 20:17:58.401700 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.399524    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"956d556026f44b19e237d54461eea3f29e9376fab24aa4aad16d5348f55964ff"} err="failed to get container status \"956d556026f44b19e237d54461eea3f29e9376fab24aa4aad16d5348f55964ff\": rpc error: code = NotFound desc = could not find container \"956d556026f44b19e237d54461eea3f29e9376fab24aa4aad16d5348f55964ff\": container with ID starting with 956d556026f44b19e237d54461eea3f29e9376fab24aa4aad16d5348f55964ff not found: ID does not exist"
Oct 30 20:17:58.401700 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.399555    5958 scope.go:122] "RemoveContainer" containerID="12b1e92d1e057f26118e0185e8988067370b9a8c83fa962073a8d7677cd5aef7"
Oct 30 20:17:58.422905 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.422812    5958 scope.go:122] "RemoveContainer" containerID="12b1e92d1e057f26118e0185e8988067370b9a8c83fa962073a8d7677cd5aef7"
Oct 30 20:17:58.423468 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:17:58.423290    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12b1e92d1e057f26118e0185e8988067370b9a8c83fa962073a8d7677cd5aef7\": container with ID starting with 12b1e92d1e057f26118e0185e8988067370b9a8c83fa962073a8d7677cd5aef7 not found: ID does not exist" containerID="12b1e92d1e057f26118e0185e8988067370b9a8c83fa962073a8d7677cd5aef7"
Oct 30 20:17:58.423468 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.423382    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12b1e92d1e057f26118e0185e8988067370b9a8c83fa962073a8d7677cd5aef7"} err="failed to get container status \"12b1e92d1e057f26118e0185e8988067370b9a8c83fa962073a8d7677cd5aef7\": rpc error: code = NotFound desc = could not find container \"12b1e92d1e057f26118e0185e8988067370b9a8c83fa962073a8d7677cd5aef7\": container with ID starting with 12b1e92d1e057f26118e0185e8988067370b9a8c83fa962073a8d7677cd5aef7 not found: ID does not exist"
Oct 30 20:17:58.466480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.466430    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.466608 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.466482    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.466608 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.466497    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.466608 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.466509    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.466608 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.466522    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.466608 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.466536    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.466608 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.466548    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7csq\" (UniqueName: \"kubernetes.io/projected/0a7169d8-4a5a-486d-840c-20a92d74bef5-kube-api-access-z7csq\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.466608 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.466563    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a7169d8-4a5a-486d-840c-20a92d74bef5-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.498837 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.494227    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"]
Oct 30 20:17:58.498837 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.498781    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-t8qgm"]
Oct 30 20:17:58.562243 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.562209    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:17:58.609531 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.609493    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"]
Oct 30 20:17:58.613206 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.613175    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-nrd6m"]
Oct 30 20:17:58.668881 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.668801    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc641430-3d40-4d0d-b2c6-b551b090da1f-kube-api-access-2c65c\" (UniqueName: \"kubernetes.io/projected/cc641430-3d40-4d0d-b2c6-b551b090da1f-kube-api-access-2c65c\") pod \"cc641430-3d40-4d0d-b2c6-b551b090da1f\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") "
Oct 30 20:17:58.669038 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.668957    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-podinfo\") pod \"cc641430-3d40-4d0d-b2c6-b551b090da1f\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") "
Oct 30 20:17:58.669094 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.669063    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-workload-socket\") pod \"cc641430-3d40-4d0d-b2c6-b551b090da1f\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") "
Oct 30 20:17:58.669094 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.669090    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-data\") pod \"cc641430-3d40-4d0d-b2c6-b551b090da1f\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") "
Oct 30 20:17:58.669190 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.669123    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc641430-3d40-4d0d-b2c6-b551b090da1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc641430-3d40-4d0d-b2c6-b551b090da1f-istiod-ca-cert\") pod \"cc641430-3d40-4d0d-b2c6-b551b090da1f\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") "
Oct 30 20:17:58.669190 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.669161    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-workload-certs\") pod \"cc641430-3d40-4d0d-b2c6-b551b090da1f\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") "
Oct 30 20:17:58.669301 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.669192    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-credential-socket\") pod \"cc641430-3d40-4d0d-b2c6-b551b090da1f\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") "
Oct 30 20:17:58.669356 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.669301    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-envoy\") pod \"cc641430-3d40-4d0d-b2c6-b551b090da1f\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") "
Oct 30 20:17:58.669413 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.669379    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-data" pod "cc641430-3d40-4d0d-b2c6-b551b090da1f" (UID: "cc641430-3d40-4d0d-b2c6-b551b090da1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.669587 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.669395    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-token\" (UniqueName: \"kubernetes.io/projected/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-token\") pod \"cc641430-3d40-4d0d-b2c6-b551b090da1f\" (UID: \"cc641430-3d40-4d0d-b2c6-b551b090da1f\") "
Oct 30 20:17:58.669587 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.669500    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-workload-socket" pod "cc641430-3d40-4d0d-b2c6-b551b090da1f" (UID: "cc641430-3d40-4d0d-b2c6-b551b090da1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.669587 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.669525    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-workload-certs" pod "cc641430-3d40-4d0d-b2c6-b551b090da1f" (UID: "cc641430-3d40-4d0d-b2c6-b551b090da1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.669762 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.669642    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc641430-3d40-4d0d-b2c6-b551b090da1f-istiod-ca-cert" pod "cc641430-3d40-4d0d-b2c6-b551b090da1f" (UID: "cc641430-3d40-4d0d-b2c6-b551b090da1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:58.669762 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.669741    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-credential-socket" pod "cc641430-3d40-4d0d-b2c6-b551b090da1f" (UID: "cc641430-3d40-4d0d-b2c6-b551b090da1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.670157 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.670137    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.670241 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.670161    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.670241 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.670175    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc641430-3d40-4d0d-b2c6-b551b090da1f-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.670241 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.670187    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.670241 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.670200    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.671645 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.671618    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-token" pod "cc641430-3d40-4d0d-b2c6-b551b090da1f" (UID: "cc641430-3d40-4d0d-b2c6-b551b090da1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:58.671811 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.671786    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-envoy" pod "cc641430-3d40-4d0d-b2c6-b551b090da1f" (UID: "cc641430-3d40-4d0d-b2c6-b551b090da1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:58.671895 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.671877    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-podinfo" pod "cc641430-3d40-4d0d-b2c6-b551b090da1f" (UID: "cc641430-3d40-4d0d-b2c6-b551b090da1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:17:58.671956 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.671893    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc641430-3d40-4d0d-b2c6-b551b090da1f-kube-api-access-2c65c" pod "cc641430-3d40-4d0d-b2c6-b551b090da1f" (UID: "cc641430-3d40-4d0d-b2c6-b551b090da1f"). InnerVolumeSpecName "kube-api-access-2c65c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:58.772041 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.771988    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.772041 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.772036    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.772312 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.772063    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c65c\" (UniqueName: \"kubernetes.io/projected/cc641430-3d40-4d0d-b2c6-b551b090da1f-kube-api-access-2c65c\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.772312 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.772078    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc641430-3d40-4d0d-b2c6-b551b090da1f-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:58.776175 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.776141    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"]
Oct 30 20:17:58.779436 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.779412    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-66mmv"]
Oct 30 20:17:58.889001 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.888960    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"]
Oct 30 20:17:58.896780 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:58.896732    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-78vjf"]
Oct 30 20:17:59.008615 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:59.008532    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"]
Oct 30 20:17:59.013536 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:59.013504    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-ddplq"]
Oct 30 20:17:59.295406 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:59.295321    5958 generic.go:356] "Generic (PLEG): container finished" podID="cc641430-3d40-4d0d-b2c6-b551b090da1f" containerID="94028ad1b4ccceb187ceb8c0143eece317dd6904d953ba6bcb74b45d5a6cfcb3" exitCode=0
Oct 30 20:17:59.295406 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:59.295375    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"
Oct 30 20:17:59.295678 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:59.295408    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89" event={"ID":"cc641430-3d40-4d0d-b2c6-b551b090da1f","Type":"ContainerDied","Data":"94028ad1b4ccceb187ceb8c0143eece317dd6904d953ba6bcb74b45d5a6cfcb3"}
Oct 30 20:17:59.295678 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:59.295564    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89" event={"ID":"cc641430-3d40-4d0d-b2c6-b551b090da1f","Type":"ContainerDied","Data":"c86b9da573ec1b2ad747f9163cb5fc892c85e7f74fc0694fc0c8a6d27d283354"}
Oct 30 20:17:59.295678 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:59.295584    5958 scope.go:122] "RemoveContainer" containerID="94028ad1b4ccceb187ceb8c0143eece317dd6904d953ba6bcb74b45d5a6cfcb3"
Oct 30 20:17:59.310294 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:59.310265    5958 scope.go:122] "RemoveContainer" containerID="94028ad1b4ccceb187ceb8c0143eece317dd6904d953ba6bcb74b45d5a6cfcb3"
Oct 30 20:17:59.310666 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:17:59.310643    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94028ad1b4ccceb187ceb8c0143eece317dd6904d953ba6bcb74b45d5a6cfcb3\": container with ID starting with 94028ad1b4ccceb187ceb8c0143eece317dd6904d953ba6bcb74b45d5a6cfcb3 not found: ID does not exist" containerID="94028ad1b4ccceb187ceb8c0143eece317dd6904d953ba6bcb74b45d5a6cfcb3"
Oct 30 20:17:59.310747 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:59.310674    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94028ad1b4ccceb187ceb8c0143eece317dd6904d953ba6bcb74b45d5a6cfcb3"} err="failed to get container status \"94028ad1b4ccceb187ceb8c0143eece317dd6904d953ba6bcb74b45d5a6cfcb3\": rpc error: code = NotFound desc = could not find container \"94028ad1b4ccceb187ceb8c0143eece317dd6904d953ba6bcb74b45d5a6cfcb3\": container with ID starting with 94028ad1b4ccceb187ceb8c0143eece317dd6904d953ba6bcb74b45d5a6cfcb3 not found: ID does not exist"
Oct 30 20:17:59.401372 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:59.401335    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a7169d8-4a5a-486d-840c-20a92d74bef5" path="/var/lib/kubelet/pods/0a7169d8-4a5a-486d-840c-20a92d74bef5/volumes"
Oct 30 20:17:59.401942 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:59.401926    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3017938e-ba45-4d82-b5bb-15919e2b413e" path="/var/lib/kubelet/pods/3017938e-ba45-4d82-b5bb-15919e2b413e/volumes"
Oct 30 20:17:59.402487 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:59.402461    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48c11b72-fdc6-4d2c-849b-136a5a53259d" path="/var/lib/kubelet/pods/48c11b72-fdc6-4d2c-849b-136a5a53259d/volumes"
Oct 30 20:17:59.403276 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:59.403259    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="551cc517-12e5-49b1-beca-2ca8348fd2b8" path="/var/lib/kubelet/pods/551cc517-12e5-49b1-beca-2ca8348fd2b8/volumes"
Oct 30 20:17:59.403788 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:59.403771    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83b30dc9-50cf-4d71-9d59-b37e1e19f971" path="/var/lib/kubelet/pods/83b30dc9-50cf-4d71-9d59-b37e1e19f971/volumes"
Oct 30 20:17:59.404228 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:59.404214    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a977973c-341c-48ec-9df5-8035da148b90" path="/var/lib/kubelet/pods/a977973c-341c-48ec-9df5-8035da148b90/volumes"
Oct 30 20:17:59.462225 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:59.462177    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"]
Oct 30 20:17:59.468122 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:17:59.468082    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8dz89"]
Oct 30 20:18:01.017025 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.016976    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89" podUID="6bea084f-db30-4c5d-9da2-91143a121dc4" containerName="payload-processing" containerID="cri-o://e4700fbb327ed6615ad65b417ada0c256106dabdf98b36a7fd14bbfcaee8be8a" gracePeriod=25
Oct 30 20:18:01.421885 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.421839    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc641430-3d40-4d0d-b2c6-b551b090da1f" path="/var/lib/kubelet/pods/cc641430-3d40-4d0d-b2c6-b551b090da1f/volumes"
Oct 30 20:18:01.816335 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.816299    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:18:01.846871 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.846831    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-data\") pod \"3b3d965b-503e-44f3-b95d-b7c599310b81\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") "
Oct 30 20:18:01.847046 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.846881    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-token\" (UniqueName: \"kubernetes.io/projected/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-token\") pod \"3b3d965b-503e-44f3-b95d-b7c599310b81\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") "
Oct 30 20:18:01.847128 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.847064    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b3d965b-503e-44f3-b95d-b7c599310b81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b3d965b-503e-44f3-b95d-b7c599310b81-istiod-ca-cert\") pod \"3b3d965b-503e-44f3-b95d-b7c599310b81\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") "
Oct 30 20:18:01.847128 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.847114    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-podinfo\") pod \"3b3d965b-503e-44f3-b95d-b7c599310b81\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") "
Oct 30 20:18:01.847279 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.847174    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-credential-socket\") pod \"3b3d965b-503e-44f3-b95d-b7c599310b81\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") "
Oct 30 20:18:01.847279 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.847180    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-data" pod "3b3d965b-503e-44f3-b95d-b7c599310b81" (UID: "3b3d965b-503e-44f3-b95d-b7c599310b81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:01.847279 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.847224    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b3d965b-503e-44f3-b95d-b7c599310b81-kube-api-access-5dtm4\" (UniqueName: \"kubernetes.io/projected/3b3d965b-503e-44f3-b95d-b7c599310b81-kube-api-access-5dtm4\") pod \"3b3d965b-503e-44f3-b95d-b7c599310b81\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") "
Oct 30 20:18:01.847445 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.847279    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-workload-socket\") pod \"3b3d965b-503e-44f3-b95d-b7c599310b81\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") "
Oct 30 20:18:01.847445 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.847377    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-envoy\") pod \"3b3d965b-503e-44f3-b95d-b7c599310b81\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") "
Oct 30 20:18:01.847573 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.847477    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-workload-certs\") pod \"3b3d965b-503e-44f3-b95d-b7c599310b81\" (UID: \"3b3d965b-503e-44f3-b95d-b7c599310b81\") "
Oct 30 20:18:01.847573 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.847553    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b3d965b-503e-44f3-b95d-b7c599310b81-istiod-ca-cert" pod "3b3d965b-503e-44f3-b95d-b7c599310b81" (UID: "3b3d965b-503e-44f3-b95d-b7c599310b81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:01.847666 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.847564    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-credential-socket" pod "3b3d965b-503e-44f3-b95d-b7c599310b81" (UID: "3b3d965b-503e-44f3-b95d-b7c599310b81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:01.847772 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.847750    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-workload-socket" pod "3b3d965b-503e-44f3-b95d-b7c599310b81" (UID: "3b3d965b-503e-44f3-b95d-b7c599310b81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:01.848042 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.848001    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-workload-certs" pod "3b3d965b-503e-44f3-b95d-b7c599310b81" (UID: "3b3d965b-503e-44f3-b95d-b7c599310b81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:01.848428 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.848395    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b3d965b-503e-44f3-b95d-b7c599310b81-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:01.848428 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.848425    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:01.848609 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.848441    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:01.848609 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.848473    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:01.848609 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.848486    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:01.849715 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.849688    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-token" pod "3b3d965b-503e-44f3-b95d-b7c599310b81" (UID: "3b3d965b-503e-44f3-b95d-b7c599310b81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:01.849819 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.849800    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-envoy" pod "3b3d965b-503e-44f3-b95d-b7c599310b81" (UID: "3b3d965b-503e-44f3-b95d-b7c599310b81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:18:01.849952 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.849917    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b3d965b-503e-44f3-b95d-b7c599310b81-kube-api-access-5dtm4" pod "3b3d965b-503e-44f3-b95d-b7c599310b81" (UID: "3b3d965b-503e-44f3-b95d-b7c599310b81"). InnerVolumeSpecName "kube-api-access-5dtm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:01.849952 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.849931    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-podinfo" pod "3b3d965b-503e-44f3-b95d-b7c599310b81" (UID: "3b3d965b-503e-44f3-b95d-b7c599310b81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:18:01.950331 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.950229    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:01.950331 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.950274    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dtm4\" (UniqueName: \"kubernetes.io/projected/3b3d965b-503e-44f3-b95d-b7c599310b81-kube-api-access-5dtm4\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:01.950331 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.950285    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:01.950331 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:01.950295    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b3d965b-503e-44f3-b95d-b7c599310b81-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:02.335679 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:02.335643    5958 generic.go:356] "Generic (PLEG): container finished" podID="3b3d965b-503e-44f3-b95d-b7c599310b81" containerID="b2e55881e5f45d47e7da5fb443c1edc0aa0abc8a738898cb087e57364b63506e" exitCode=0
Oct 30 20:18:02.336096 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:02.335694    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"
Oct 30 20:18:02.336096 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:02.335725    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4" event={"ID":"3b3d965b-503e-44f3-b95d-b7c599310b81","Type":"ContainerDied","Data":"b2e55881e5f45d47e7da5fb443c1edc0aa0abc8a738898cb087e57364b63506e"}
Oct 30 20:18:02.336096 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:02.335788    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4" event={"ID":"3b3d965b-503e-44f3-b95d-b7c599310b81","Type":"ContainerDied","Data":"8247d47a9a06d74fcc6be1204e1a3258ef6d6f3e913ae413e2c7429457444e7f"}
Oct 30 20:18:02.336096 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:02.335812    5958 scope.go:122] "RemoveContainer" containerID="b2e55881e5f45d47e7da5fb443c1edc0aa0abc8a738898cb087e57364b63506e"
Oct 30 20:18:02.348444 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:02.348422    5958 scope.go:122] "RemoveContainer" containerID="b2e55881e5f45d47e7da5fb443c1edc0aa0abc8a738898cb087e57364b63506e"
Oct 30 20:18:02.348805 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:18:02.348786    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e55881e5f45d47e7da5fb443c1edc0aa0abc8a738898cb087e57364b63506e\": container with ID starting with b2e55881e5f45d47e7da5fb443c1edc0aa0abc8a738898cb087e57364b63506e not found: ID does not exist" containerID="b2e55881e5f45d47e7da5fb443c1edc0aa0abc8a738898cb087e57364b63506e"
Oct 30 20:18:02.348869 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:02.348815    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e55881e5f45d47e7da5fb443c1edc0aa0abc8a738898cb087e57364b63506e"} err="failed to get container status \"b2e55881e5f45d47e7da5fb443c1edc0aa0abc8a738898cb087e57364b63506e\": rpc error: code = NotFound desc = could not find container \"b2e55881e5f45d47e7da5fb443c1edc0aa0abc8a738898cb087e57364b63506e\": container with ID starting with b2e55881e5f45d47e7da5fb443c1edc0aa0abc8a738898cb087e57364b63506e not found: ID does not exist"
Oct 30 20:18:02.502843 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:02.502803    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"]
Oct 30 20:18:02.553366 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:02.553321    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-w8zf4"]
Oct 30 20:18:02.726444 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:02.726358    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89" podUID="6bea084f-db30-4c5d-9da2-91143a121dc4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:18:03.400967 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:03.400930    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b3d965b-503e-44f3-b95d-b7c599310b81" path="/var/lib/kubelet/pods/3b3d965b-503e-44f3-b95d-b7c599310b81/volumes"
Oct 30 20:18:06.614870 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:18:06.614828    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:18:12.725809 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:12.725770    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89" podUID="6bea084f-db30-4c5d-9da2-91143a121dc4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:18:21.293147 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:21.293076    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89"
Oct 30 20:18:21.390942 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:21.390907    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bea084f-db30-4c5d-9da2-91143a121dc4-kube-api-access-w6n4c\" (UniqueName: \"kubernetes.io/projected/6bea084f-db30-4c5d-9da2-91143a121dc4-kube-api-access-w6n4c\") pod \"6bea084f-db30-4c5d-9da2-91143a121dc4\" (UID: \"6bea084f-db30-4c5d-9da2-91143a121dc4\") "
Oct 30 20:18:21.391148 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:21.391129    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bea084f-db30-4c5d-9da2-91143a121dc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bea084f-db30-4c5d-9da2-91143a121dc4-plugins-config-volume\") pod \"6bea084f-db30-4c5d-9da2-91143a121dc4\" (UID: \"6bea084f-db30-4c5d-9da2-91143a121dc4\") "
Oct 30 20:18:21.391575 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:21.391546    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bea084f-db30-4c5d-9da2-91143a121dc4-plugins-config-volume" pod "6bea084f-db30-4c5d-9da2-91143a121dc4" (UID: "6bea084f-db30-4c5d-9da2-91143a121dc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:21.391785 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:21.391767    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bea084f-db30-4c5d-9da2-91143a121dc4-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:21.393324 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:21.393300    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bea084f-db30-4c5d-9da2-91143a121dc4-kube-api-access-w6n4c" pod "6bea084f-db30-4c5d-9da2-91143a121dc4" (UID: "6bea084f-db30-4c5d-9da2-91143a121dc4"). InnerVolumeSpecName "kube-api-access-w6n4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:21.484405 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:21.484368    5958 generic.go:356] "Generic (PLEG): container finished" podID="6bea084f-db30-4c5d-9da2-91143a121dc4" containerID="e4700fbb327ed6615ad65b417ada0c256106dabdf98b36a7fd14bbfcaee8be8a" exitCode=0
Oct 30 20:18:21.484608 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:21.484415    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89"
Oct 30 20:18:21.484608 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:21.484483    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89" event={"ID":"6bea084f-db30-4c5d-9da2-91143a121dc4","Type":"ContainerDied","Data":"e4700fbb327ed6615ad65b417ada0c256106dabdf98b36a7fd14bbfcaee8be8a"}
Oct 30 20:18:21.484608 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:21.484538    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89" event={"ID":"6bea084f-db30-4c5d-9da2-91143a121dc4","Type":"ContainerDied","Data":"9c02792d41fc250d52d521154964ca8f6eca406b7c1b5fec434582fa8d199f8a"}
Oct 30 20:18:21.484608 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:21.484560    5958 scope.go:122] "RemoveContainer" containerID="e4700fbb327ed6615ad65b417ada0c256106dabdf98b36a7fd14bbfcaee8be8a"
Oct 30 20:18:21.493445 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:21.493414    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6n4c\" (UniqueName: \"kubernetes.io/projected/6bea084f-db30-4c5d-9da2-91143a121dc4-kube-api-access-w6n4c\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:18:21.495524 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:21.495502    5958 scope.go:122] "RemoveContainer" containerID="e4700fbb327ed6615ad65b417ada0c256106dabdf98b36a7fd14bbfcaee8be8a"
Oct 30 20:18:21.495794 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:18:21.495773    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4700fbb327ed6615ad65b417ada0c256106dabdf98b36a7fd14bbfcaee8be8a\": container with ID starting with e4700fbb327ed6615ad65b417ada0c256106dabdf98b36a7fd14bbfcaee8be8a not found: ID does not exist" containerID="e4700fbb327ed6615ad65b417ada0c256106dabdf98b36a7fd14bbfcaee8be8a"
Oct 30 20:18:21.495846 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:21.495802    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4700fbb327ed6615ad65b417ada0c256106dabdf98b36a7fd14bbfcaee8be8a"} err="failed to get container status \"e4700fbb327ed6615ad65b417ada0c256106dabdf98b36a7fd14bbfcaee8be8a\": rpc error: code = NotFound desc = could not find container \"e4700fbb327ed6615ad65b417ada0c256106dabdf98b36a7fd14bbfcaee8be8a\": container with ID starting with e4700fbb327ed6615ad65b417ada0c256106dabdf98b36a7fd14bbfcaee8be8a not found: ID does not exist"
Oct 30 20:18:21.652981 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:21.652934    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89"]
Oct 30 20:18:21.659187 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:21.659160    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-37732de7-6df6b6d844-9wx89"]
Oct 30 20:18:23.402856 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:18:23.402821    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bea084f-db30-4c5d-9da2-91143a121dc4" path="/var/lib/kubelet/pods/6bea084f-db30-4c5d-9da2-91143a121dc4/volumes"
Oct 30 20:18:36.610415 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:18:36.610364    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:19:06.608054 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:19:06.607968    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:19:10.130788 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:10.130748    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6"]
Oct 30 20:19:15.132061 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:15.132009    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6" podUID="8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f" containerName="payload-pre-processing" containerID="cri-o://56f9a820b840978404faaae17a5816eba7097ea66626c7f298fbcf87fbd1c46f" gracePeriod=25
Oct 30 20:19:15.461907 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:15.461872    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6"
Oct 30 20:19:15.574106 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:15.574068    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f-plugins-config-volume\") pod \"8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f\" (UID: \"8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f\") "
Oct 30 20:19:15.574287 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:15.574184    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f-kube-api-access-fgzqq\" (UniqueName: \"kubernetes.io/projected/8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f-kube-api-access-fgzqq\") pod \"8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f\" (UID: \"8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f\") "
Oct 30 20:19:15.574545 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:15.574522    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f-plugins-config-volume" pod "8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f" (UID: "8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:19:15.574899 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:15.574873    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:15.576670 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:15.576633    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f-kube-api-access-fgzqq" pod "8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f" (UID: "8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f"). InnerVolumeSpecName "kube-api-access-fgzqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:15.676578 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:15.676473    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgzqq\" (UniqueName: \"kubernetes.io/projected/8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f-kube-api-access-fgzqq\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:15.981621 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:15.981523    5958 generic.go:356] "Generic (PLEG): container finished" podID="8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f" containerID="56f9a820b840978404faaae17a5816eba7097ea66626c7f298fbcf87fbd1c46f" exitCode=0
Oct 30 20:19:15.981996 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:15.981971    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6" event={"ID":"8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f","Type":"ContainerDied","Data":"56f9a820b840978404faaae17a5816eba7097ea66626c7f298fbcf87fbd1c46f"}
Oct 30 20:19:15.982177 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:15.982157    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6" event={"ID":"8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f","Type":"ContainerDied","Data":"62dc0e7eb25e02e888af79fbdc3a7bdc65217ec28e2160e77d54f3b6ae9b04f8"}
Oct 30 20:19:15.982298 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:15.982284    5958 scope.go:122] "RemoveContainer" containerID="56f9a820b840978404faaae17a5816eba7097ea66626c7f298fbcf87fbd1c46f"
Oct 30 20:19:15.986494 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:15.982436    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6"
Oct 30 20:19:16.012169 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:16.012139    5958 scope.go:122] "RemoveContainer" containerID="56f9a820b840978404faaae17a5816eba7097ea66626c7f298fbcf87fbd1c46f"
Oct 30 20:19:16.012583 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:19:16.012558    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56f9a820b840978404faaae17a5816eba7097ea66626c7f298fbcf87fbd1c46f\": container with ID starting with 56f9a820b840978404faaae17a5816eba7097ea66626c7f298fbcf87fbd1c46f not found: ID does not exist" containerID="56f9a820b840978404faaae17a5816eba7097ea66626c7f298fbcf87fbd1c46f"
Oct 30 20:19:16.012666 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:16.012596    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56f9a820b840978404faaae17a5816eba7097ea66626c7f298fbcf87fbd1c46f"} err="failed to get container status \"56f9a820b840978404faaae17a5816eba7097ea66626c7f298fbcf87fbd1c46f\": rpc error: code = NotFound desc = could not find container \"56f9a820b840978404faaae17a5816eba7097ea66626c7f298fbcf87fbd1c46f\": container with ID starting with 56f9a820b840978404faaae17a5816eba7097ea66626c7f298fbcf87fbd1c46f not found: ID does not exist"
Oct 30 20:19:16.157523 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:16.157471    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6"]
Oct 30 20:19:16.178394 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:16.178353    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f44ec2-69fb895d79-h27g6"]
Oct 30 20:19:17.400342 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:17.400306    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f" path="/var/lib/kubelet/pods/8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f/volumes"
Oct 30 20:19:36.607971 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:19:36.607930    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:19:37.814428 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:37.814387    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"]
Oct 30 20:19:37.814853 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:37.814610    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk" podUID="7c2eca7f-c86c-427e-b1c5-1d33ded45e42" containerName="istio-proxy" containerID="cri-o://40fc7497374a4d34d0d94b9b1d14f7f5cff7d1e0e7f7404088d46fb1f3899744" gracePeriod=30
Oct 30 20:19:41.441895 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.441862    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"]
Oct 30 20:19:41.442743 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442721    5958 cpu_manager.go:414] "containerMap: removing container" podUID="0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9" containerName="istio-proxy"
Oct 30 20:19:41.442743 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442744    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9" containerName="istio-proxy"
Oct 30 20:19:41.442851 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442759    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c35c95a6-d115-43cf-a332-19dc91c7bdd4" containerName="payload-pre-processing"
Oct 30 20:19:41.442851 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442764    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c35c95a6-d115-43cf-a332-19dc91c7bdd4" containerName="payload-pre-processing"
Oct 30 20:19:41.442851 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442772    5958 cpu_manager.go:414] "containerMap: removing container" podUID="6bea084f-db30-4c5d-9da2-91143a121dc4" containerName="payload-processing"
Oct 30 20:19:41.442851 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442779    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bea084f-db30-4c5d-9da2-91143a121dc4" containerName="payload-processing"
Oct 30 20:19:41.442851 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442789    5958 cpu_manager.go:414] "containerMap: removing container" podUID="e20ab886-1287-453f-baea-c74e65e5ccad" containerName="cleanup"
Oct 30 20:19:41.442851 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442794    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e20ab886-1287-453f-baea-c74e65e5ccad" containerName="cleanup"
Oct 30 20:19:41.442851 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442802    5958 cpu_manager.go:414] "containerMap: removing container" podUID="e20ab886-1287-453f-baea-c74e65e5ccad" containerName="cleanup"
Oct 30 20:19:41.442851 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442807    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e20ab886-1287-453f-baea-c74e65e5ccad" containerName="cleanup"
Oct 30 20:19:41.442851 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442817    5958 cpu_manager.go:414] "containerMap: removing container" podUID="20541f8a-ed94-4723-adc8-f6ee759ca566" containerName="payload-processing"
Oct 30 20:19:41.442851 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442822    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20541f8a-ed94-4723-adc8-f6ee759ca566" containerName="payload-processing"
Oct 30 20:19:41.442851 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442829    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ed65872e-55ca-416c-bf1d-05e1bc0047b5" containerName="cleanup"
Oct 30 20:19:41.442851 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442833    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed65872e-55ca-416c-bf1d-05e1bc0047b5" containerName="cleanup"
Oct 30 20:19:41.442851 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442840    5958 cpu_manager.go:414] "containerMap: removing container" podUID="551cc517-12e5-49b1-beca-2ca8348fd2b8" containerName="istio-proxy"
Oct 30 20:19:41.442851 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442844    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551cc517-12e5-49b1-beca-2ca8348fd2b8" containerName="istio-proxy"
Oct 30 20:19:41.442851 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442851    5958 cpu_manager.go:414] "containerMap: removing container" podUID="83b30dc9-50cf-4d71-9d59-b37e1e19f971" containerName="istio-proxy"
Oct 30 20:19:41.442851 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442856    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b30dc9-50cf-4d71-9d59-b37e1e19f971" containerName="istio-proxy"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442862    5958 cpu_manager.go:414] "containerMap: removing container" podUID="a820cb88-eb76-459c-bb08-28837aa714a1" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442866    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a820cb88-eb76-459c-bb08-28837aa714a1" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442872    5958 cpu_manager.go:414] "containerMap: removing container" podUID="a820cb88-eb76-459c-bb08-28837aa714a1" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442875    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a820cb88-eb76-459c-bb08-28837aa714a1" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442885    5958 cpu_manager.go:414] "containerMap: removing container" podUID="e8847b33-95a0-4136-9d77-82b9008d9d7d" containerName="istio-proxy"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442891    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8847b33-95a0-4136-9d77-82b9008d9d7d" containerName="istio-proxy"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442900    5958 cpu_manager.go:414] "containerMap: removing container" podUID="76eb3bb9-b9b8-4804-a31b-18165a69ce7f" containerName="istio-proxy"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442904    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76eb3bb9-b9b8-4804-a31b-18165a69ce7f" containerName="istio-proxy"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442911    5958 cpu_manager.go:414] "containerMap: removing container" podUID="6a6506dc-92f6-46cd-8e96-030a02603d96" containerName="maas-api"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442915    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a6506dc-92f6-46cd-8e96-030a02603d96" containerName="maas-api"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442923    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ea2b023d-3aaf-47f7-8fa2-385d0ab430a5" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442927    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2b023d-3aaf-47f7-8fa2-385d0ab430a5" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442933    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ea2b023d-3aaf-47f7-8fa2-385d0ab430a5" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442937    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2b023d-3aaf-47f7-8fa2-385d0ab430a5" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442944    5958 cpu_manager.go:414] "containerMap: removing container" podUID="b7fae60d-346d-4d4a-8851-82586d2ddf65" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442948    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7fae60d-346d-4d4a-8851-82586d2ddf65" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442956    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3017938e-ba45-4d82-b5bb-15919e2b413e" containerName="istio-proxy"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442959    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3017938e-ba45-4d82-b5bb-15919e2b413e" containerName="istio-proxy"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442967    5958 cpu_manager.go:414] "containerMap: removing container" podUID="e20ab886-1287-453f-baea-c74e65e5ccad" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442971    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e20ab886-1287-453f-baea-c74e65e5ccad" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442979    5958 cpu_manager.go:414] "containerMap: removing container" podUID="a977973c-341c-48ec-9df5-8035da148b90" containerName="istio-proxy"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442983    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a977973c-341c-48ec-9df5-8035da148b90" containerName="istio-proxy"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442989    5958 cpu_manager.go:414] "containerMap: removing container" podUID="cc641430-3d40-4d0d-b2c6-b551b090da1f" containerName="istio-proxy"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442993    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc641430-3d40-4d0d-b2c6-b551b090da1f" containerName="istio-proxy"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.442998    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ea2b023d-3aaf-47f7-8fa2-385d0ab430a5" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443002    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2b023d-3aaf-47f7-8fa2-385d0ab430a5" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443009    5958 cpu_manager.go:414] "containerMap: removing container" podUID="5c2103c1-53ee-4aaf-ab8e-75c5c93390a1" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443013    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2103c1-53ee-4aaf-ab8e-75c5c93390a1" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443019    5958 cpu_manager.go:414] "containerMap: removing container" podUID="b7fae60d-346d-4d4a-8851-82586d2ddf65" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443023    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7fae60d-346d-4d4a-8851-82586d2ddf65" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443034    5958 cpu_manager.go:414] "containerMap: removing container" podUID="0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2" containerName="istio-proxy"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443038    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2" containerName="istio-proxy"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443044    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3b3d965b-503e-44f3-b95d-b7c599310b81" containerName="istio-proxy"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443048    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b3d965b-503e-44f3-b95d-b7c599310b81" containerName="istio-proxy"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443059    5958 cpu_manager.go:414] "containerMap: removing container" podUID="5c2103c1-53ee-4aaf-ab8e-75c5c93390a1" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443063    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2103c1-53ee-4aaf-ab8e-75c5c93390a1" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443075    5958 cpu_manager.go:414] "containerMap: removing container" podUID="73694a08-761d-412e-9a30-5c02be8900dd" containerName="payload-pre-processing"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443079    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73694a08-761d-412e-9a30-5c02be8900dd" containerName="payload-pre-processing"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443086    5958 cpu_manager.go:414] "containerMap: removing container" podUID="8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f" containerName="payload-pre-processing"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443091    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f" containerName="payload-pre-processing"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443100    5958 cpu_manager.go:414] "containerMap: removing container" podUID="b7fae60d-346d-4d4a-8851-82586d2ddf65" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443105    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7fae60d-346d-4d4a-8851-82586d2ddf65" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443113    5958 cpu_manager.go:414] "containerMap: removing container" podUID="5c2103c1-53ee-4aaf-ab8e-75c5c93390a1" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443117    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2103c1-53ee-4aaf-ab8e-75c5c93390a1" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443127    5958 cpu_manager.go:414] "containerMap: removing container" podUID="48c11b72-fdc6-4d2c-849b-136a5a53259d" containerName="istio-proxy"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443133    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48c11b72-fdc6-4d2c-849b-136a5a53259d" containerName="istio-proxy"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443144    5958 cpu_manager.go:414] "containerMap: removing container" podUID="0a7169d8-4a5a-486d-840c-20a92d74bef5" containerName="istio-proxy"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443148    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a7169d8-4a5a-486d-840c-20a92d74bef5" containerName="istio-proxy"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443156    5958 cpu_manager.go:414] "containerMap: removing container" podUID="f03388cb-31a5-41c1-9997-116ec2be50d9" containerName="payload-pre-processing"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443160    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f03388cb-31a5-41c1-9997-116ec2be50d9" containerName="payload-pre-processing"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443167    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ed65872e-55ca-416c-bf1d-05e1bc0047b5" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443171    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed65872e-55ca-416c-bf1d-05e1bc0047b5" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443261    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7fae60d-346d-4d4a-8851-82586d2ddf65" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443269    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2103c1-53ee-4aaf-ab8e-75c5c93390a1" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443276    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="48c11b72-fdc6-4d2c-849b-136a5a53259d" containerName="istio-proxy"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443284    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a7169d8-4a5a-486d-840c-20a92d74bef5" containerName="istio-proxy"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443289    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="f03388cb-31a5-41c1-9997-116ec2be50d9" containerName="payload-pre-processing"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443295    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed65872e-55ca-416c-bf1d-05e1bc0047b5" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443301    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="a820cb88-eb76-459c-bb08-28837aa714a1" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443314    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b2fbafd-bdd0-48b0-bdd9-73b84d6951f9" containerName="istio-proxy"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443319    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c35c95a6-d115-43cf-a332-19dc91c7bdd4" containerName="payload-pre-processing"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443326    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bea084f-db30-4c5d-9da2-91143a121dc4" containerName="payload-processing"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443332    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="e20ab886-1287-453f-baea-c74e65e5ccad" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443337    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="e20ab886-1287-453f-baea-c74e65e5ccad" containerName="cleanup"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443344    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="20541f8a-ed94-4723-adc8-f6ee759ca566" containerName="payload-processing"
Oct 30 20:19:41.443315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443349    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed65872e-55ca-416c-bf1d-05e1bc0047b5" containerName="cleanup"
Oct 30 20:19:41.445537 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443353    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="551cc517-12e5-49b1-beca-2ca8348fd2b8" containerName="istio-proxy"
Oct 30 20:19:41.445537 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443359    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b30dc9-50cf-4d71-9d59-b37e1e19f971" containerName="istio-proxy"
Oct 30 20:19:41.445537 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443365    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="a820cb88-eb76-459c-bb08-28837aa714a1" containerName="cleanup"
Oct 30 20:19:41.445537 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443369    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="a820cb88-eb76-459c-bb08-28837aa714a1" containerName="cleanup"
Oct 30 20:19:41.445537 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443374    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8847b33-95a0-4136-9d77-82b9008d9d7d" containerName="istio-proxy"
Oct 30 20:19:41.445537 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443379    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="76eb3bb9-b9b8-4804-a31b-18165a69ce7f" containerName="istio-proxy"
Oct 30 20:19:41.445537 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443384    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a6506dc-92f6-46cd-8e96-030a02603d96" containerName="maas-api"
Oct 30 20:19:41.445537 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443390    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2b023d-3aaf-47f7-8fa2-385d0ab430a5" containerName="cleanup"
Oct 30 20:19:41.445537 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443395    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2b023d-3aaf-47f7-8fa2-385d0ab430a5" containerName="cleanup"
Oct 30 20:19:41.445537 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443400    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7fae60d-346d-4d4a-8851-82586d2ddf65" containerName="cleanup"
Oct 30 20:19:41.445537 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443405    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3017938e-ba45-4d82-b5bb-15919e2b413e" containerName="istio-proxy"
Oct 30 20:19:41.445537 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443409    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="e20ab886-1287-453f-baea-c74e65e5ccad" containerName="cleanup"
Oct 30 20:19:41.445537 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443416    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="a977973c-341c-48ec-9df5-8035da148b90" containerName="istio-proxy"
Oct 30 20:19:41.445537 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443422    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc641430-3d40-4d0d-b2c6-b551b090da1f" containerName="istio-proxy"
Oct 30 20:19:41.445537 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443426    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2b023d-3aaf-47f7-8fa2-385d0ab430a5" containerName="cleanup"
Oct 30 20:19:41.445537 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443433    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a44ce4e-0ecf-4b57-b400-abef5d8f2dd2" containerName="istio-proxy"
Oct 30 20:19:41.445537 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443438    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b3d965b-503e-44f3-b95d-b7c599310b81" containerName="istio-proxy"
Oct 30 20:19:41.445537 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443444    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed65872e-55ca-416c-bf1d-05e1bc0047b5" containerName="cleanup"
Oct 30 20:19:41.445537 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443467    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2103c1-53ee-4aaf-ab8e-75c5c93390a1" containerName="cleanup"
Oct 30 20:19:41.445537 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443478    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="73694a08-761d-412e-9a30-5c02be8900dd" containerName="payload-pre-processing"
Oct 30 20:19:41.445537 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.443487    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b11fbdf-f4e7-4403-b4c5-eb73dc1f034f" containerName="payload-pre-processing"
Oct 30 20:19:41.447033 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.447016    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.623939 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.623898    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.624114 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.623967    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.624114 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.624061    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.624114 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.624092    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.624219 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.624120    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.624301 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.624275    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8tjz\" (UniqueName: \"kubernetes.io/projected/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-kube-api-access-v8tjz\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.624396 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.624384    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.624434 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.624425    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.624487 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.624467    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.659152 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.659111    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"]
Oct 30 20:19:41.725424 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.725340    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.725424 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.725402    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.725637 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.725536    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8tjz\" (UniqueName: \"kubernetes.io/projected/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-kube-api-access-v8tjz\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.725675 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.725652    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.725712 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.725685    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.725851 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.725826    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.725945 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.725872    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.726002 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.725983    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.726057 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.726015    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.726111 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.726082    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.726189 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.726164    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.726356 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.726330    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.726512 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.726445    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.726794 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.726776    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.727813 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.727791    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.728369 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.728351    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.786957 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.786920    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:41.787107 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:41.787063    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8tjz\" (UniqueName: \"kubernetes.io/projected/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-kube-api-access-v8tjz\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z2qnt\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:42.062240 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:42.062167    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:42.265148 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:42.265112    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:19:42.265312 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:42.265179    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk" podUID="7c2eca7f-c86c-427e-b1c5-1d33ded45e42" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:19:42.495600 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:42.495570    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"]
Oct 30 20:19:42.497263 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:19:42.497233    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc76c29_f2ac_42be_8f69_7f3ac0f5869f.slice/crio-5eeff01c3e9dd92650b6789fb1b0c9fa7807949a0fbe0ff75aa71706c70e21a6 WatchSource:0}: Error finding container 5eeff01c3e9dd92650b6789fb1b0c9fa7807949a0fbe0ff75aa71706c70e21a6: Status 404 returned error can't find the container with id 5eeff01c3e9dd92650b6789fb1b0c9fa7807949a0fbe0ff75aa71706c70e21a6
Oct 30 20:19:42.499989 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:42.499953    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:19:42.500069 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:42.500022    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:19:42.500069 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:42.500051    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:19:43.099679 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.099606    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:19:43.215265 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.215223    5958 generic.go:356] "Generic (PLEG): container finished" podID="7c2eca7f-c86c-427e-b1c5-1d33ded45e42" containerID="40fc7497374a4d34d0d94b9b1d14f7f5cff7d1e0e7f7404088d46fb1f3899744" exitCode=0
Oct 30 20:19:43.215480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.215276    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk" event={"ID":"7c2eca7f-c86c-427e-b1c5-1d33ded45e42","Type":"ContainerDied","Data":"40fc7497374a4d34d0d94b9b1d14f7f5cff7d1e0e7f7404088d46fb1f3899744"}
Oct 30 20:19:43.215480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.215375    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk" event={"ID":"7c2eca7f-c86c-427e-b1c5-1d33ded45e42","Type":"ContainerDied","Data":"c9af10c6606634bb14ecb1b44a31104b05837d9fc1888e0947cd3c3c6fe6b500"}
Oct 30 20:19:43.215480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.215286    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"
Oct 30 20:19:43.215480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.215411    5958 scope.go:122] "RemoveContainer" containerID="40fc7497374a4d34d0d94b9b1d14f7f5cff7d1e0e7f7404088d46fb1f3899744"
Oct 30 20:19:43.216940 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.216921    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt" event={"ID":"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f","Type":"ContainerStarted","Data":"651f422b69acf6b753f6d4e983d6413b128ff72ed82b95460743d7fea4af45db"}
Oct 30 20:19:43.217007 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.216957    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt" event={"ID":"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f","Type":"ContainerStarted","Data":"5eeff01c3e9dd92650b6789fb1b0c9fa7807949a0fbe0ff75aa71706c70e21a6"}
Oct 30 20:19:43.232245 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.232201    5958 scope.go:122] "RemoveContainer" containerID="40fc7497374a4d34d0d94b9b1d14f7f5cff7d1e0e7f7404088d46fb1f3899744"
Oct 30 20:19:43.232841 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:19:43.232804    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40fc7497374a4d34d0d94b9b1d14f7f5cff7d1e0e7f7404088d46fb1f3899744\": container with ID starting with 40fc7497374a4d34d0d94b9b1d14f7f5cff7d1e0e7f7404088d46fb1f3899744 not found: ID does not exist" containerID="40fc7497374a4d34d0d94b9b1d14f7f5cff7d1e0e7f7404088d46fb1f3899744"
Oct 30 20:19:43.232978 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.232865    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40fc7497374a4d34d0d94b9b1d14f7f5cff7d1e0e7f7404088d46fb1f3899744"} err="failed to get container status \"40fc7497374a4d34d0d94b9b1d14f7f5cff7d1e0e7f7404088d46fb1f3899744\": rpc error: code = NotFound desc = could not find container \"40fc7497374a4d34d0d94b9b1d14f7f5cff7d1e0e7f7404088d46fb1f3899744\": container with ID starting with 40fc7497374a4d34d0d94b9b1d14f7f5cff7d1e0e7f7404088d46fb1f3899744 not found: ID does not exist"
Oct 30 20:19:43.267524 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.262033    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-workload-socket\") pod \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") "
Oct 30 20:19:43.267699 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.267430    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-workload-socket" pod "7c2eca7f-c86c-427e-b1c5-1d33ded45e42" (UID: "7c2eca7f-c86c-427e-b1c5-1d33ded45e42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:19:43.267699 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.267592    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istiod-ca-cert\") pod \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") "
Oct 30 20:19:43.267825 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.267733    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-kube-api-access-v7fc5\" (UniqueName: \"kubernetes.io/projected/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-kube-api-access-v7fc5\") pod \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") "
Oct 30 20:19:43.267825 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.267798    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-workload-certs\") pod \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") "
Oct 30 20:19:43.267938 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.267870    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istiod-ca-cert" pod "7c2eca7f-c86c-427e-b1c5-1d33ded45e42" (UID: "7c2eca7f-c86c-427e-b1c5-1d33ded45e42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:19:43.267938 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.267872    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-token\" (UniqueName: \"kubernetes.io/projected/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-token\") pod \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") "
Oct 30 20:19:43.267938 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.267935    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-envoy\") pod \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") "
Oct 30 20:19:43.268087 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.267996    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-podinfo\") pod \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") "
Oct 30 20:19:43.268087 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.268036    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-credential-socket\") pod \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") "
Oct 30 20:19:43.268087 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.268063    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-data\") pod \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\" (UID: \"7c2eca7f-c86c-427e-b1c5-1d33ded45e42\") "
Oct 30 20:19:43.268641 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.268623    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:43.270675 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.270518    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:43.270675 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.269045    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-data" pod "7c2eca7f-c86c-427e-b1c5-1d33ded45e42" (UID: "7c2eca7f-c86c-427e-b1c5-1d33ded45e42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:19:43.270675 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.269487    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-workload-certs" pod "7c2eca7f-c86c-427e-b1c5-1d33ded45e42" (UID: "7c2eca7f-c86c-427e-b1c5-1d33ded45e42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:19:43.270675 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.269767    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-credential-socket" pod "7c2eca7f-c86c-427e-b1c5-1d33ded45e42" (UID: "7c2eca7f-c86c-427e-b1c5-1d33ded45e42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:19:43.270966 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.270935    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-kube-api-access-v7fc5" pod "7c2eca7f-c86c-427e-b1c5-1d33ded45e42" (UID: "7c2eca7f-c86c-427e-b1c5-1d33ded45e42"). InnerVolumeSpecName "kube-api-access-v7fc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:43.271142 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.271119    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-token" pod "7c2eca7f-c86c-427e-b1c5-1d33ded45e42" (UID: "7c2eca7f-c86c-427e-b1c5-1d33ded45e42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:43.272619 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.272589    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-podinfo" pod "7c2eca7f-c86c-427e-b1c5-1d33ded45e42" (UID: "7c2eca7f-c86c-427e-b1c5-1d33ded45e42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:19:43.273261 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.273230    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-envoy" pod "7c2eca7f-c86c-427e-b1c5-1d33ded45e42" (UID: "7c2eca7f-c86c-427e-b1c5-1d33ded45e42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:19:43.353635 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.353518    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt" podStartSLOduration=2.35349871 podStartE2EDuration="2.35349871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:43.350590663 +0000 UTC m=+2222.082806054" watchObservedRunningTime="2026-10-30 20:19:43.35349871 +0000 UTC m=+2222.085714102"
Oct 30 20:19:43.373143 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.373099    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7fc5\" (UniqueName: \"kubernetes.io/projected/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-kube-api-access-v7fc5\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:43.373143 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.373143    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:43.373409 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.373160    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:43.373409 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.373177    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:43.373409 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.373191    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:43.373409 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.373205    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:43.373409 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.373220    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c2eca7f-c86c-427e-b1c5-1d33ded45e42-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:43.657532 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.652529    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"]
Oct 30 20:19:43.662984 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.662942    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-k96dk"]
Oct 30 20:19:43.702893 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.702849    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7"]
Oct 30 20:19:43.813680 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:43.813640    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g"]
Oct 30 20:19:44.070861 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:44.069726    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:44.071654 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:44.071615    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:19:44.071903 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:44.071685    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt" podUID="3bc76c29-f2ac-42be-8f69-7f3ac0f5869f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 20:19:45.062601 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:45.062559    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:19:45.063016 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:45.062626    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt" podUID="3bc76c29-f2ac-42be-8f69-7f3ac0f5869f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 20:19:45.405904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:45.403674    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c2eca7f-c86c-427e-b1c5-1d33ded45e42" path="/var/lib/kubelet/pods/7c2eca7f-c86c-427e-b1c5-1d33ded45e42/volumes"
Oct 30 20:19:46.064070 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:46.064032    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:19:46.064589 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:46.064100    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt" podUID="3bc76c29-f2ac-42be-8f69-7f3ac0f5869f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 20:19:47.069236 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:47.068836    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:19:47.069236 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:47.068895    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt" podUID="3bc76c29-f2ac-42be-8f69-7f3ac0f5869f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 20:19:48.062658 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:48.062605    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:19:48.062854 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:48.062694    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt" podUID="3bc76c29-f2ac-42be-8f69-7f3ac0f5869f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 20:19:48.703190 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:48.703135    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7" podUID="fc5eb38c-c314-4a32-846f-d5b63f2ad009" containerName="payload-processing" containerID="cri-o://42583b5c359fd37e713f7b451ca4d2773358a7451fdf6e8133c8b14e55f02cc4" gracePeriod=25
Oct 30 20:19:48.815173 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:48.814931    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g" podUID="be48f946-80c5-464d-bd18-408c6d239046" containerName="payload-pre-processing" containerID="cri-o://a735774fd912e669f85d760567391a6ebb5fdaa378124d5b867b82236885ddec" gracePeriod=25
Oct 30 20:19:49.032084 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.032051    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7"
Oct 30 20:19:49.063211 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.063176    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:19:49.063381 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.063249    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt" podUID="3bc76c29-f2ac-42be-8f69-7f3ac0f5869f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 20:19:49.114639 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.114598    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc5eb38c-c314-4a32-846f-d5b63f2ad009-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5eb38c-c314-4a32-846f-d5b63f2ad009-plugins-config-volume\") pod \"fc5eb38c-c314-4a32-846f-d5b63f2ad009\" (UID: \"fc5eb38c-c314-4a32-846f-d5b63f2ad009\") "
Oct 30 20:19:49.114889 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.114875    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc5eb38c-c314-4a32-846f-d5b63f2ad009-kube-api-access-vkwdm\" (UniqueName: \"kubernetes.io/projected/fc5eb38c-c314-4a32-846f-d5b63f2ad009-kube-api-access-vkwdm\") pod \"fc5eb38c-c314-4a32-846f-d5b63f2ad009\" (UID: \"fc5eb38c-c314-4a32-846f-d5b63f2ad009\") "
Oct 30 20:19:49.114997 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.114973    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5eb38c-c314-4a32-846f-d5b63f2ad009-plugins-config-volume" pod "fc5eb38c-c314-4a32-846f-d5b63f2ad009" (UID: "fc5eb38c-c314-4a32-846f-d5b63f2ad009"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:19:49.115487 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.115427    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5eb38c-c314-4a32-846f-d5b63f2ad009-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:49.117337 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.117307    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5eb38c-c314-4a32-846f-d5b63f2ad009-kube-api-access-vkwdm" pod "fc5eb38c-c314-4a32-846f-d5b63f2ad009" (UID: "fc5eb38c-c314-4a32-846f-d5b63f2ad009"). InnerVolumeSpecName "kube-api-access-vkwdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:49.138434 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.138401    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g"
Oct 30 20:19:49.216492 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.216439    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be48f946-80c5-464d-bd18-408c6d239046-kube-api-access-lnswf\" (UniqueName: \"kubernetes.io/projected/be48f946-80c5-464d-bd18-408c6d239046-kube-api-access-lnswf\") pod \"be48f946-80c5-464d-bd18-408c6d239046\" (UID: \"be48f946-80c5-464d-bd18-408c6d239046\") "
Oct 30 20:19:49.216695 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.216666    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be48f946-80c5-464d-bd18-408c6d239046-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be48f946-80c5-464d-bd18-408c6d239046-plugins-config-volume\") pod \"be48f946-80c5-464d-bd18-408c6d239046\" (UID: \"be48f946-80c5-464d-bd18-408c6d239046\") "
Oct 30 20:19:49.217069 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.217042    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be48f946-80c5-464d-bd18-408c6d239046-plugins-config-volume" pod "be48f946-80c5-464d-bd18-408c6d239046" (UID: "be48f946-80c5-464d-bd18-408c6d239046"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:19:49.217321 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.217304    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be48f946-80c5-464d-bd18-408c6d239046-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:49.217382 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.217326    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkwdm\" (UniqueName: \"kubernetes.io/projected/fc5eb38c-c314-4a32-846f-d5b63f2ad009-kube-api-access-vkwdm\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:49.218935 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.218903    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be48f946-80c5-464d-bd18-408c6d239046-kube-api-access-lnswf" pod "be48f946-80c5-464d-bd18-408c6d239046" (UID: "be48f946-80c5-464d-bd18-408c6d239046"). InnerVolumeSpecName "kube-api-access-lnswf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:49.312215 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.312113    5958 generic.go:356] "Generic (PLEG): container finished" podID="be48f946-80c5-464d-bd18-408c6d239046" containerID="a735774fd912e669f85d760567391a6ebb5fdaa378124d5b867b82236885ddec" exitCode=0
Oct 30 20:19:49.312392 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.312332    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g"
Oct 30 20:19:49.313579 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.313549    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g" event={"ID":"be48f946-80c5-464d-bd18-408c6d239046","Type":"ContainerDied","Data":"a735774fd912e669f85d760567391a6ebb5fdaa378124d5b867b82236885ddec"}
Oct 30 20:19:49.313823 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.313789    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g" event={"ID":"be48f946-80c5-464d-bd18-408c6d239046","Type":"ContainerDied","Data":"81161ec3a767d2e30b3a65c803a4613a6342265e276d140a3c595ffb2387ec26"}
Oct 30 20:19:49.314365 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.314345    5958 scope.go:122] "RemoveContainer" containerID="a735774fd912e669f85d760567391a6ebb5fdaa378124d5b867b82236885ddec"
Oct 30 20:19:49.319183 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.319142    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnswf\" (UniqueName: \"kubernetes.io/projected/be48f946-80c5-464d-bd18-408c6d239046-kube-api-access-lnswf\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:49.328030 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.327985    5958 generic.go:356] "Generic (PLEG): container finished" podID="fc5eb38c-c314-4a32-846f-d5b63f2ad009" containerID="42583b5c359fd37e713f7b451ca4d2773358a7451fdf6e8133c8b14e55f02cc4" exitCode=0
Oct 30 20:19:49.328265 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.328232    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7" event={"ID":"fc5eb38c-c314-4a32-846f-d5b63f2ad009","Type":"ContainerDied","Data":"42583b5c359fd37e713f7b451ca4d2773358a7451fdf6e8133c8b14e55f02cc4"}
Oct 30 20:19:49.328342 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.328310    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7" event={"ID":"fc5eb38c-c314-4a32-846f-d5b63f2ad009","Type":"ContainerDied","Data":"f998643467e14f34c80a0bc2904620f8960d11ab30934e1d96317b6314f3f944"}
Oct 30 20:19:49.328564 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.328397    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7"
Oct 30 20:19:49.339113 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.339029    5958 scope.go:122] "RemoveContainer" containerID="a735774fd912e669f85d760567391a6ebb5fdaa378124d5b867b82236885ddec"
Oct 30 20:19:49.339717 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:19:49.339677    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a735774fd912e669f85d760567391a6ebb5fdaa378124d5b867b82236885ddec\": container with ID starting with a735774fd912e669f85d760567391a6ebb5fdaa378124d5b867b82236885ddec not found: ID does not exist" containerID="a735774fd912e669f85d760567391a6ebb5fdaa378124d5b867b82236885ddec"
Oct 30 20:19:49.339927 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.339739    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a735774fd912e669f85d760567391a6ebb5fdaa378124d5b867b82236885ddec"} err="failed to get container status \"a735774fd912e669f85d760567391a6ebb5fdaa378124d5b867b82236885ddec\": rpc error: code = NotFound desc = could not find container \"a735774fd912e669f85d760567391a6ebb5fdaa378124d5b867b82236885ddec\": container with ID starting with a735774fd912e669f85d760567391a6ebb5fdaa378124d5b867b82236885ddec not found: ID does not exist"
Oct 30 20:19:49.339927 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.339769    5958 scope.go:122] "RemoveContainer" containerID="42583b5c359fd37e713f7b451ca4d2773358a7451fdf6e8133c8b14e55f02cc4"
Oct 30 20:19:49.356746 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.356717    5958 scope.go:122] "RemoveContainer" containerID="42583b5c359fd37e713f7b451ca4d2773358a7451fdf6e8133c8b14e55f02cc4"
Oct 30 20:19:49.357207 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:19:49.357178    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42583b5c359fd37e713f7b451ca4d2773358a7451fdf6e8133c8b14e55f02cc4\": container with ID starting with 42583b5c359fd37e713f7b451ca4d2773358a7451fdf6e8133c8b14e55f02cc4 not found: ID does not exist" containerID="42583b5c359fd37e713f7b451ca4d2773358a7451fdf6e8133c8b14e55f02cc4"
Oct 30 20:19:49.357308 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.357215    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42583b5c359fd37e713f7b451ca4d2773358a7451fdf6e8133c8b14e55f02cc4"} err="failed to get container status \"42583b5c359fd37e713f7b451ca4d2773358a7451fdf6e8133c8b14e55f02cc4\": rpc error: code = NotFound desc = could not find container \"42583b5c359fd37e713f7b451ca4d2773358a7451fdf6e8133c8b14e55f02cc4\": container with ID starting with 42583b5c359fd37e713f7b451ca4d2773358a7451fdf6e8133c8b14e55f02cc4 not found: ID does not exist"
Oct 30 20:19:49.482770 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.482726    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g"]
Oct 30 20:19:49.489794 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.489760    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-770dfe-666788c67f-z4k2g"]
Oct 30 20:19:49.599065 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.599029    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7"]
Oct 30 20:19:49.601838 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:49.601811    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-770dfe-79db6f4b7d-2mjk7"]
Oct 30 20:19:50.070919 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:50.070888    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 20:19:50.071367 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:50.070945    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt" podUID="3bc76c29-f2ac-42be-8f69-7f3ac0f5869f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 20:19:51.225081 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:51.225034    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:51.227649 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:51.226949    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:51.228438 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:51.228409    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:19:51.403414 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:51.403375    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be48f946-80c5-464d-bd18-408c6d239046" path="/var/lib/kubelet/pods/be48f946-80c5-464d-bd18-408c6d239046/volumes"
Oct 30 20:19:51.403885 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:51.403871    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc5eb38c-c314-4a32-846f-d5b63f2ad009" path="/var/lib/kubelet/pods/fc5eb38c-c314-4a32-846f-d5b63f2ad009/volumes"
Oct 30 20:19:53.379982 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:53.379950    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"]
Oct 30 20:19:53.429257 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:19:53.429211    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"]
Oct 30 20:20:00.093375 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:00.093328    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"]
Oct 30 20:20:00.093937 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:00.093634    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd" podUID="23b32875-225b-469d-8794-c47053a71ebc" containerName="istio-proxy" containerID="cri-o://857e436ff122c97e535b830567b55a6436e9ab3a89cd8e56f9cc9c39539da25e" gracePeriod=30
Oct 30 20:20:05.371883 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.371818    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:20:05.468271 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.468236    5958 generic.go:356] "Generic (PLEG): container finished" podID="23b32875-225b-469d-8794-c47053a71ebc" containerID="857e436ff122c97e535b830567b55a6436e9ab3a89cd8e56f9cc9c39539da25e" exitCode=0
Oct 30 20:20:05.468476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.468294    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"
Oct 30 20:20:05.468476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.468315    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd" event={"ID":"23b32875-225b-469d-8794-c47053a71ebc","Type":"ContainerDied","Data":"857e436ff122c97e535b830567b55a6436e9ab3a89cd8e56f9cc9c39539da25e"}
Oct 30 20:20:05.468476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.468370    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd" event={"ID":"23b32875-225b-469d-8794-c47053a71ebc","Type":"ContainerDied","Data":"5266f30ba77f87f1bfcad198f118c23a5918cce85ee04f733dbab578001c55fd"}
Oct 30 20:20:05.468476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.468394    5958 scope.go:122] "RemoveContainer" containerID="857e436ff122c97e535b830567b55a6436e9ab3a89cd8e56f9cc9c39539da25e"
Oct 30 20:20:05.479213 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.479191    5958 scope.go:122] "RemoveContainer" containerID="ab3eb8f380d8b23cc73f3c5f1a7a37c6559614a6200a59a9635ebce44917ebb6"
Oct 30 20:20:05.489194 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.489173    5958 scope.go:122] "RemoveContainer" containerID="857e436ff122c97e535b830567b55a6436e9ab3a89cd8e56f9cc9c39539da25e"
Oct 30 20:20:05.489544 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:20:05.489521    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857e436ff122c97e535b830567b55a6436e9ab3a89cd8e56f9cc9c39539da25e\": container with ID starting with 857e436ff122c97e535b830567b55a6436e9ab3a89cd8e56f9cc9c39539da25e not found: ID does not exist" containerID="857e436ff122c97e535b830567b55a6436e9ab3a89cd8e56f9cc9c39539da25e"
Oct 30 20:20:05.489619 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.489551    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857e436ff122c97e535b830567b55a6436e9ab3a89cd8e56f9cc9c39539da25e"} err="failed to get container status \"857e436ff122c97e535b830567b55a6436e9ab3a89cd8e56f9cc9c39539da25e\": rpc error: code = NotFound desc = could not find container \"857e436ff122c97e535b830567b55a6436e9ab3a89cd8e56f9cc9c39539da25e\": container with ID starting with 857e436ff122c97e535b830567b55a6436e9ab3a89cd8e56f9cc9c39539da25e not found: ID does not exist"
Oct 30 20:20:05.489619 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.489572    5958 scope.go:122] "RemoveContainer" containerID="ab3eb8f380d8b23cc73f3c5f1a7a37c6559614a6200a59a9635ebce44917ebb6"
Oct 30 20:20:05.489837 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:20:05.489819    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab3eb8f380d8b23cc73f3c5f1a7a37c6559614a6200a59a9635ebce44917ebb6\": container with ID starting with ab3eb8f380d8b23cc73f3c5f1a7a37c6559614a6200a59a9635ebce44917ebb6 not found: ID does not exist" containerID="ab3eb8f380d8b23cc73f3c5f1a7a37c6559614a6200a59a9635ebce44917ebb6"
Oct 30 20:20:05.489889 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.489842    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab3eb8f380d8b23cc73f3c5f1a7a37c6559614a6200a59a9635ebce44917ebb6"} err="failed to get container status \"ab3eb8f380d8b23cc73f3c5f1a7a37c6559614a6200a59a9635ebce44917ebb6\": rpc error: code = NotFound desc = could not find container \"ab3eb8f380d8b23cc73f3c5f1a7a37c6559614a6200a59a9635ebce44917ebb6\": container with ID starting with ab3eb8f380d8b23cc73f3c5f1a7a37c6559614a6200a59a9635ebce44917ebb6 not found: ID does not exist"
Oct 30 20:20:05.497371 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.497344    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-workload-socket\") pod \"23b32875-225b-469d-8794-c47053a71ebc\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") "
Oct 30 20:20:05.497492 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.497431    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23b32875-225b-469d-8794-c47053a71ebc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23b32875-225b-469d-8794-c47053a71ebc-istio-podinfo\") pod \"23b32875-225b-469d-8794-c47053a71ebc\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") "
Oct 30 20:20:05.497492 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.497483    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23b32875-225b-469d-8794-c47053a71ebc-kube-api-access-rtpbf\" (UniqueName: \"kubernetes.io/projected/23b32875-225b-469d-8794-c47053a71ebc-kube-api-access-rtpbf\") pod \"23b32875-225b-469d-8794-c47053a71ebc\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") "
Oct 30 20:20:05.497612 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.497587    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-credential-socket\") pod \"23b32875-225b-469d-8794-c47053a71ebc\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") "
Oct 30 20:20:05.497667 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.497645    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-workload-certs\") pod \"23b32875-225b-469d-8794-c47053a71ebc\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") "
Oct 30 20:20:05.497721 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.497660    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-workload-socket" pod "23b32875-225b-469d-8794-c47053a71ebc" (UID: "23b32875-225b-469d-8794-c47053a71ebc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:05.497721 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.497684    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23b32875-225b-469d-8794-c47053a71ebc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23b32875-225b-469d-8794-c47053a71ebc-istiod-ca-cert\") pod \"23b32875-225b-469d-8794-c47053a71ebc\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") "
Oct 30 20:20:05.497867 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.497849    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-istio-data\") pod \"23b32875-225b-469d-8794-c47053a71ebc\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") "
Oct 30 20:20:05.497935 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.497876    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-credential-socket" pod "23b32875-225b-469d-8794-c47053a71ebc" (UID: "23b32875-225b-469d-8794-c47053a71ebc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:05.497935 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.497887    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-workload-certs" pod "23b32875-225b-469d-8794-c47053a71ebc" (UID: "23b32875-225b-469d-8794-c47053a71ebc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:05.498007 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.497954    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-istio-envoy\") pod \"23b32875-225b-469d-8794-c47053a71ebc\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") "
Oct 30 20:20:05.498007 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.497962    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b32875-225b-469d-8794-c47053a71ebc-istiod-ca-cert" pod "23b32875-225b-469d-8794-c47053a71ebc" (UID: "23b32875-225b-469d-8794-c47053a71ebc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:05.498093 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.498046    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23b32875-225b-469d-8794-c47053a71ebc-istio-token\" (UniqueName: \"kubernetes.io/projected/23b32875-225b-469d-8794-c47053a71ebc-istio-token\") pod \"23b32875-225b-469d-8794-c47053a71ebc\" (UID: \"23b32875-225b-469d-8794-c47053a71ebc\") "
Oct 30 20:20:05.498143 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.498119    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-istio-data" pod "23b32875-225b-469d-8794-c47053a71ebc" (UID: "23b32875-225b-469d-8794-c47053a71ebc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:05.499001 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.498970    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:05.499138 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.499004    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:05.499138 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.499019    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:05.499138 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.499033    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23b32875-225b-469d-8794-c47053a71ebc-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:05.499138 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.499046    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:05.500036 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.500007    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23b32875-225b-469d-8794-c47053a71ebc-istio-podinfo" pod "23b32875-225b-469d-8794-c47053a71ebc" (UID: "23b32875-225b-469d-8794-c47053a71ebc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:20:05.500139 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.500098    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b32875-225b-469d-8794-c47053a71ebc-kube-api-access-rtpbf" pod "23b32875-225b-469d-8794-c47053a71ebc" (UID: "23b32875-225b-469d-8794-c47053a71ebc"). InnerVolumeSpecName "kube-api-access-rtpbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:05.500184 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.500149    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-istio-envoy" pod "23b32875-225b-469d-8794-c47053a71ebc" (UID: "23b32875-225b-469d-8794-c47053a71ebc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:05.500374 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.500352    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b32875-225b-469d-8794-c47053a71ebc-istio-token" pod "23b32875-225b-469d-8794-c47053a71ebc" (UID: "23b32875-225b-469d-8794-c47053a71ebc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:05.600106 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.600062    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23b32875-225b-469d-8794-c47053a71ebc-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:05.600106 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.600098    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtpbf\" (UniqueName: \"kubernetes.io/projected/23b32875-225b-469d-8794-c47053a71ebc-kube-api-access-rtpbf\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:05.600106 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.600110    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23b32875-225b-469d-8794-c47053a71ebc-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:05.600368 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.600120    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23b32875-225b-469d-8794-c47053a71ebc-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:05.893285 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.893251    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"]
Oct 30 20:20:05.895977 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:05.895941    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-f44ec2-openshift-default-68884b6975-fhfrd"]
Oct 30 20:20:06.608351 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:20:06.608313    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:20:07.399713 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:07.399676    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23b32875-225b-469d-8794-c47053a71ebc" path="/var/lib/kubelet/pods/23b32875-225b-469d-8794-c47053a71ebc/volumes"
Oct 30 20:20:08.402510 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.402467    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf" podUID="a1551c50-4d89-4227-838c-4644c41de49b" containerName="main" containerID="cri-o://1a22a35f291863cd1c671bcc21a1ab8893c990f2217ed611a81fba31ea4a6c91" gracePeriod=45
Oct 30 20:20:08.451542 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.451478    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main" containerID="cri-o://c9e70d47d0fa20467a1fcaff683f5fd6422dd47ec24f1d979b23542a31354833" gracePeriod=45
Oct 30 20:20:08.734557 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.734515    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"]
Oct 30 20:20:08.734788 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.734764    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr" podUID="b8b98921-b374-47ca-93d1-6ccd66cae325" containerName="maas-api" containerID="cri-o://054be163d5c50ff24e5873d8df6d6c523e4fb3f85c95672473e4a64df673dfb3" gracePeriod=30
Oct 30 20:20:08.818650 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.818562    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:20:08.824535 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.824509    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 20:20:08.948605 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.948569    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-home\" (UniqueName: \"kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-home\") pod \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\" (UID: \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\") "
Oct 30 20:20:08.948820 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.948630    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1551c50-4d89-4227-838c-4644c41de49b-tls-certs\" (UniqueName: \"kubernetes.io/secret/a1551c50-4d89-4227-838c-4644c41de49b-tls-certs\") pod \"a1551c50-4d89-4227-838c-4644c41de49b\" (UID: \"a1551c50-4d89-4227-838c-4644c41de49b\") "
Oct 30 20:20:08.948820 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.948743    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-kube-api-access-4bdpt\" (UniqueName: \"kubernetes.io/projected/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-kube-api-access-4bdpt\") pod \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\" (UID: \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\") "
Oct 30 20:20:08.948820 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.948785    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1551c50-4d89-4227-838c-4644c41de49b-kube-api-access-vz6r5\" (UniqueName: \"kubernetes.io/projected/a1551c50-4d89-4227-838c-4644c41de49b-kube-api-access-vz6r5\") pod \"a1551c50-4d89-4227-838c-4644c41de49b\" (UID: \"a1551c50-4d89-4227-838c-4644c41de49b\") "
Oct 30 20:20:08.948820 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.948816    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-dshm\") pod \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\" (UID: \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\") "
Oct 30 20:20:08.949036 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.948835    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-tmp-dir\") pod \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\" (UID: \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\") "
Oct 30 20:20:08.949036 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.948864    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-home" pod "d1929aa5-fcd4-4609-8cbc-68afb34bdadc" (UID: "d1929aa5-fcd4-4609-8cbc-68afb34bdadc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:08.949036 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.948914    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-home\" (UniqueName: \"kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-home\") pod \"a1551c50-4d89-4227-838c-4644c41de49b\" (UID: \"a1551c50-4d89-4227-838c-4644c41de49b\") "
Oct 30 20:20:08.949036 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.948980    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-tmp-dir\") pod \"a1551c50-4d89-4227-838c-4644c41de49b\" (UID: \"a1551c50-4d89-4227-838c-4644c41de49b\") "
Oct 30 20:20:08.949036 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.949031    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-model-cache\") pod \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\" (UID: \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\") "
Oct 30 20:20:08.949321 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.949087    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-model-cache\") pod \"a1551c50-4d89-4227-838c-4644c41de49b\" (UID: \"a1551c50-4d89-4227-838c-4644c41de49b\") "
Oct 30 20:20:08.949321 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.949139    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-dshm\") pod \"a1551c50-4d89-4227-838c-4644c41de49b\" (UID: \"a1551c50-4d89-4227-838c-4644c41de49b\") "
Oct 30 20:20:08.949321 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.949200    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-tls-certs\") pod \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\" (UID: \"d1929aa5-fcd4-4609-8cbc-68afb34bdadc\") "
Oct 30 20:20:08.949813 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.949515    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-tmp-dir" pod "d1929aa5-fcd4-4609-8cbc-68afb34bdadc" (UID: "d1929aa5-fcd4-4609-8cbc-68afb34bdadc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:08.949813 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.949368    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-home" pod "a1551c50-4d89-4227-838c-4644c41de49b" (UID: "a1551c50-4d89-4227-838c-4644c41de49b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:08.949813 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.949761    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-tmp-dir" pod "a1551c50-4d89-4227-838c-4644c41de49b" (UID: "a1551c50-4d89-4227-838c-4644c41de49b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:08.949813 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.949767    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-model-cache" pod "d1929aa5-fcd4-4609-8cbc-68afb34bdadc" (UID: "d1929aa5-fcd4-4609-8cbc-68afb34bdadc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:08.950118 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.949847    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-model-cache" pod "a1551c50-4d89-4227-838c-4644c41de49b" (UID: "a1551c50-4d89-4227-838c-4644c41de49b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:08.950220 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.950037    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-home\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:08.950315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.950235    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-tmp-dir\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:08.952092 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.951395    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-home\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:08.952092 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.951420    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-tmp-dir\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:08.952092 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.951433    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-model-cache\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:08.952092 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.951478    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-model-cache\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:08.952621 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.952584    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-kube-api-access-4bdpt" pod "d1929aa5-fcd4-4609-8cbc-68afb34bdadc" (UID: "d1929aa5-fcd4-4609-8cbc-68afb34bdadc"). InnerVolumeSpecName "kube-api-access-4bdpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:08.953695 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.953659    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1551c50-4d89-4227-838c-4644c41de49b-tls-certs" pod "a1551c50-4d89-4227-838c-4644c41de49b" (UID: "a1551c50-4d89-4227-838c-4644c41de49b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:08.953849 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.953823    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-tls-certs" pod "d1929aa5-fcd4-4609-8cbc-68afb34bdadc" (UID: "d1929aa5-fcd4-4609-8cbc-68afb34bdadc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:08.953928 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.953878    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-dshm" pod "a1551c50-4d89-4227-838c-4644c41de49b" (UID: "a1551c50-4d89-4227-838c-4644c41de49b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:08.953982 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.953965    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-dshm" pod "d1929aa5-fcd4-4609-8cbc-68afb34bdadc" (UID: "d1929aa5-fcd4-4609-8cbc-68afb34bdadc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:08.954749 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:08.954720    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1551c50-4d89-4227-838c-4644c41de49b-kube-api-access-vz6r5" pod "a1551c50-4d89-4227-838c-4644c41de49b" (UID: "a1551c50-4d89-4227-838c-4644c41de49b"). InnerVolumeSpecName "kube-api-access-vz6r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:09.015357 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.015194    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"
Oct 30 20:20:09.052656 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.052620    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1551c50-4d89-4227-838c-4644c41de49b-tls-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:09.052656 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.052655    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bdpt\" (UniqueName: \"kubernetes.io/projected/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-kube-api-access-4bdpt\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:09.052866 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.052666    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz6r5\" (UniqueName: \"kubernetes.io/projected/a1551c50-4d89-4227-838c-4644c41de49b-kube-api-access-vz6r5\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:09.052866 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.052675    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-dshm\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:09.052866 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.052684    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1551c50-4d89-4227-838c-4644c41de49b-dshm\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:09.052866 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.052691    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1929aa5-fcd4-4609-8cbc-68afb34bdadc-tls-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:09.153773 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.153735    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b98921-b374-47ca-93d1-6ccd66cae325-kube-api-access-hjkn6\" (UniqueName: \"kubernetes.io/projected/b8b98921-b374-47ca-93d1-6ccd66cae325-kube-api-access-hjkn6\") pod \"b8b98921-b374-47ca-93d1-6ccd66cae325\" (UID: \"b8b98921-b374-47ca-93d1-6ccd66cae325\") "
Oct 30 20:20:09.153968 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.153941    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8b98921-b374-47ca-93d1-6ccd66cae325-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8b98921-b374-47ca-93d1-6ccd66cae325-metrics-certs\") pod \"b8b98921-b374-47ca-93d1-6ccd66cae325\" (UID: \"b8b98921-b374-47ca-93d1-6ccd66cae325\") "
Oct 30 20:20:09.154046 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.154032    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8b98921-b374-47ca-93d1-6ccd66cae325-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b98921-b374-47ca-93d1-6ccd66cae325-openshift-service-ca\") pod \"b8b98921-b374-47ca-93d1-6ccd66cae325\" (UID: \"b8b98921-b374-47ca-93d1-6ccd66cae325\") "
Oct 30 20:20:09.154120 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.154103    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8b98921-b374-47ca-93d1-6ccd66cae325-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b98921-b374-47ca-93d1-6ccd66cae325-maas-api-tls\") pod \"b8b98921-b374-47ca-93d1-6ccd66cae325\" (UID: \"b8b98921-b374-47ca-93d1-6ccd66cae325\") "
Oct 30 20:20:09.154600 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.154569    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b98921-b374-47ca-93d1-6ccd66cae325-openshift-service-ca" pod "b8b98921-b374-47ca-93d1-6ccd66cae325" (UID: "b8b98921-b374-47ca-93d1-6ccd66cae325"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:09.154765 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.154741    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b98921-b374-47ca-93d1-6ccd66cae325-openshift-service-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:09.156548 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.156519    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b98921-b374-47ca-93d1-6ccd66cae325-metrics-certs" pod "b8b98921-b374-47ca-93d1-6ccd66cae325" (UID: "b8b98921-b374-47ca-93d1-6ccd66cae325"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:09.156850 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.156824    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b98921-b374-47ca-93d1-6ccd66cae325-kube-api-access-hjkn6" pod "b8b98921-b374-47ca-93d1-6ccd66cae325" (UID: "b8b98921-b374-47ca-93d1-6ccd66cae325"). InnerVolumeSpecName "kube-api-access-hjkn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:09.157292 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.157263    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b98921-b374-47ca-93d1-6ccd66cae325-maas-api-tls" pod "b8b98921-b374-47ca-93d1-6ccd66cae325" (UID: "b8b98921-b374-47ca-93d1-6ccd66cae325"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:09.257343 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.257257    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjkn6\" (UniqueName: \"kubernetes.io/projected/b8b98921-b374-47ca-93d1-6ccd66cae325-kube-api-access-hjkn6\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:09.257343 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.257294    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8b98921-b374-47ca-93d1-6ccd66cae325-metrics-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:09.257343 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.257313    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b98921-b374-47ca-93d1-6ccd66cae325-maas-api-tls\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:09.497926 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.497893    5958 generic.go:356] "Generic (PLEG): container finished" podID="b8b98921-b374-47ca-93d1-6ccd66cae325" containerID="054be163d5c50ff24e5873d8df6d6c523e4fb3f85c95672473e4a64df673dfb3" exitCode=0
Oct 30 20:20:09.498397 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.497941    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"
Oct 30 20:20:09.498397 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.497981    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr" event={"ID":"b8b98921-b374-47ca-93d1-6ccd66cae325","Type":"ContainerDied","Data":"054be163d5c50ff24e5873d8df6d6c523e4fb3f85c95672473e4a64df673dfb3"}
Oct 30 20:20:09.498397 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.498050    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr" event={"ID":"b8b98921-b374-47ca-93d1-6ccd66cae325","Type":"ContainerDied","Data":"5d6135e453f15d55d70dfc695e4c67ef0b559d1cb1ed612b23cd8fca6fc42c37"}
Oct 30 20:20:09.498397 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.498089    5958 scope.go:122] "RemoveContainer" containerID="054be163d5c50ff24e5873d8df6d6c523e4fb3f85c95672473e4a64df673dfb3"
Oct 30 20:20:09.499663 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.499492    5958 generic.go:356] "Generic (PLEG): container finished" podID="a1551c50-4d89-4227-838c-4644c41de49b" containerID="1a22a35f291863cd1c671bcc21a1ab8893c990f2217ed611a81fba31ea4a6c91" exitCode=0
Oct 30 20:20:09.499663 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.499640    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf" event={"ID":"a1551c50-4d89-4227-838c-4644c41de49b","Type":"ContainerDied","Data":"1a22a35f291863cd1c671bcc21a1ab8893c990f2217ed611a81fba31ea4a6c91"}
Oct 30 20:20:09.499786 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.499671    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf" event={"ID":"a1551c50-4d89-4227-838c-4644c41de49b","Type":"ContainerDied","Data":"97ad019c6b736b5e4d66e02294f7d65e143759def33749cc43041acbaf293d8b"}
Oct 30 20:20:09.499786 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.499724    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"
Oct 30 20:20:09.502262 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.502237    5958 generic.go:356] "Generic (PLEG): container finished" podID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerID="c9e70d47d0fa20467a1fcaff683f5fd6422dd47ec24f1d979b23542a31354833" exitCode=0
Oct 30 20:20:09.502482 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.502430    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" event={"ID":"d1929aa5-fcd4-4609-8cbc-68afb34bdadc","Type":"ContainerDied","Data":"c9e70d47d0fa20467a1fcaff683f5fd6422dd47ec24f1d979b23542a31354833"}
Oct 30 20:20:09.502595 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.502501    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk" event={"ID":"d1929aa5-fcd4-4609-8cbc-68afb34bdadc","Type":"ContainerDied","Data":"7aaff9259a95f34e21404f7dd3e3fd6c3039b1ff23ce327d19c3ffcbbd3a7033"}
Oct 30 20:20:09.502595 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.502558    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"
Oct 30 20:20:09.510335 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.510315    5958 scope.go:122] "RemoveContainer" containerID="054be163d5c50ff24e5873d8df6d6c523e4fb3f85c95672473e4a64df673dfb3"
Oct 30 20:20:09.510635 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:20:09.510614    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"054be163d5c50ff24e5873d8df6d6c523e4fb3f85c95672473e4a64df673dfb3\": container with ID starting with 054be163d5c50ff24e5873d8df6d6c523e4fb3f85c95672473e4a64df673dfb3 not found: ID does not exist" containerID="054be163d5c50ff24e5873d8df6d6c523e4fb3f85c95672473e4a64df673dfb3"
Oct 30 20:20:09.510718 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.510646    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"054be163d5c50ff24e5873d8df6d6c523e4fb3f85c95672473e4a64df673dfb3"} err="failed to get container status \"054be163d5c50ff24e5873d8df6d6c523e4fb3f85c95672473e4a64df673dfb3\": rpc error: code = NotFound desc = could not find container \"054be163d5c50ff24e5873d8df6d6c523e4fb3f85c95672473e4a64df673dfb3\": container with ID starting with 054be163d5c50ff24e5873d8df6d6c523e4fb3f85c95672473e4a64df673dfb3 not found: ID does not exist"
Oct 30 20:20:09.510718 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.510665    5958 scope.go:122] "RemoveContainer" containerID="1a22a35f291863cd1c671bcc21a1ab8893c990f2217ed611a81fba31ea4a6c91"
Oct 30 20:20:09.519466 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.519430    5958 scope.go:122] "RemoveContainer" containerID="1a22a35f291863cd1c671bcc21a1ab8893c990f2217ed611a81fba31ea4a6c91"
Oct 30 20:20:09.519748 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:20:09.519727    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a22a35f291863cd1c671bcc21a1ab8893c990f2217ed611a81fba31ea4a6c91\": container with ID starting with 1a22a35f291863cd1c671bcc21a1ab8893c990f2217ed611a81fba31ea4a6c91 not found: ID does not exist" containerID="1a22a35f291863cd1c671bcc21a1ab8893c990f2217ed611a81fba31ea4a6c91"
Oct 30 20:20:09.519819 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.519754    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a22a35f291863cd1c671bcc21a1ab8893c990f2217ed611a81fba31ea4a6c91"} err="failed to get container status \"1a22a35f291863cd1c671bcc21a1ab8893c990f2217ed611a81fba31ea4a6c91\": rpc error: code = NotFound desc = could not find container \"1a22a35f291863cd1c671bcc21a1ab8893c990f2217ed611a81fba31ea4a6c91\": container with ID starting with 1a22a35f291863cd1c671bcc21a1ab8893c990f2217ed611a81fba31ea4a6c91 not found: ID does not exist"
Oct 30 20:20:09.519819 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.519769    5958 scope.go:122] "RemoveContainer" containerID="c9e70d47d0fa20467a1fcaff683f5fd6422dd47ec24f1d979b23542a31354833"
Oct 30 20:20:09.528552 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.528537    5958 scope.go:122] "RemoveContainer" containerID="c9e70d47d0fa20467a1fcaff683f5fd6422dd47ec24f1d979b23542a31354833"
Oct 30 20:20:09.528821 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:20:09.528797    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9e70d47d0fa20467a1fcaff683f5fd6422dd47ec24f1d979b23542a31354833\": container with ID starting with c9e70d47d0fa20467a1fcaff683f5fd6422dd47ec24f1d979b23542a31354833 not found: ID does not exist" containerID="c9e70d47d0fa20467a1fcaff683f5fd6422dd47ec24f1d979b23542a31354833"
Oct 30 20:20:09.528895 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.528826    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9e70d47d0fa20467a1fcaff683f5fd6422dd47ec24f1d979b23542a31354833"} err="failed to get container status \"c9e70d47d0fa20467a1fcaff683f5fd6422dd47ec24f1d979b23542a31354833\": rpc error: code = NotFound desc = could not find container \"c9e70d47d0fa20467a1fcaff683f5fd6422dd47ec24f1d979b23542a31354833\": container with ID starting with c9e70d47d0fa20467a1fcaff683f5fd6422dd47ec24f1d979b23542a31354833 not found: ID does not exist"
Oct 30 20:20:09.642584 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.642527    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf" podUID="a1551c50-4d89-4227-838c-4644c41de49b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:20:09.664684 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.664646    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"]
Oct 30 20:20:09.672657 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.672624    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f84d85/simulator-w3-f84d85-kserve-b8d9dbf57-rk9lk"]
Oct 30 20:20:09.784593 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.784507    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"]
Oct 30 20:20:09.788542 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.788504    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f84d85/distinct-2-w3-f84d85-kserve-5846c87596-xrrkf"]
Oct 30 20:20:09.909476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.908504    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"]
Oct 30 20:20:09.931504 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:09.928703    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-770dfe-f9d9cc4f6-mrhvr"]
Oct 30 20:20:11.399946 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:11.399909    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1551c50-4d89-4227-838c-4644c41de49b" path="/var/lib/kubelet/pods/a1551c50-4d89-4227-838c-4644c41de49b/volumes"
Oct 30 20:20:11.400396 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:11.400380    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8b98921-b374-47ca-93d1-6ccd66cae325" path="/var/lib/kubelet/pods/b8b98921-b374-47ca-93d1-6ccd66cae325/volumes"
Oct 30 20:20:11.400840 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:11.400825    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" path="/var/lib/kubelet/pods/d1929aa5-fcd4-4609-8cbc-68afb34bdadc/volumes"
Oct 30 20:20:23.400633 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:23.400555    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"]
Oct 30 20:20:23.401080 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:23.400732    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv" podUID="54a80c44-b6f8-44bc-9a7d-86a43f48f7c7" containerName="istio-proxy" containerID="cri-o://037fa331b2b7055496b4a412573cedb6bb8b548e78af4530d27cd1a203f583dd" gracePeriod=30
Oct 30 20:20:24.011914 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:24.011882    5958 patch_prober.go:29] interesting pod/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:20:24.012075 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:24.011947    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv" podUID="54a80c44-b6f8-44bc-9a7d-86a43f48f7c7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:20:24.179290 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:24.179259    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"]
Oct 30 20:20:24.179516 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:24.179491    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675" podUID="8bfda83c-7138-4974-ae1c-67fa552ac430" containerName="istio-proxy" containerID="cri-o://fbea68500be3cc29b5255741f5efb36efa28803ca2f10d75c30ac94f9bfd9455" gracePeriod=30
Oct 30 20:20:28.638088 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.638052    5958 generic.go:356] "Generic (PLEG): container finished" podID="54a80c44-b6f8-44bc-9a7d-86a43f48f7c7" containerID="037fa331b2b7055496b4a412573cedb6bb8b548e78af4530d27cd1a203f583dd" exitCode=0
Oct 30 20:20:28.638444 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.638129    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv" event={"ID":"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7","Type":"ContainerDied","Data":"037fa331b2b7055496b4a412573cedb6bb8b548e78af4530d27cd1a203f583dd"}
Oct 30 20:20:28.670032 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.669977    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:20:28.722831 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.722806    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-workload-socket\") pod \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") "
Oct 30 20:20:28.722955 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.722837    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-token\" (UniqueName: \"kubernetes.io/projected/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-token\") pod \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") "
Oct 30 20:20:28.722955 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.722861    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-data\") pod \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") "
Oct 30 20:20:28.722955 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.722893    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istiod-ca-cert\") pod \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") "
Oct 30 20:20:28.723114 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.722944    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-workload-certs\") pod \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") "
Oct 30 20:20:28.723114 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.723045    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-podinfo\") pod \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") "
Oct 30 20:20:28.723114 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.723085    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-workload-socket" pod "54a80c44-b6f8-44bc-9a7d-86a43f48f7c7" (UID: "54a80c44-b6f8-44bc-9a7d-86a43f48f7c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:28.723263 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.723150    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-kube-api-access-q9ss5\" (UniqueName: \"kubernetes.io/projected/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-kube-api-access-q9ss5\") pod \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") "
Oct 30 20:20:28.723263 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.723185    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-envoy\") pod \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") "
Oct 30 20:20:28.723263 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.723208    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-credential-socket\") pod \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\" (UID: \"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7\") "
Oct 30 20:20:28.723263 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.723211    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-data" pod "54a80c44-b6f8-44bc-9a7d-86a43f48f7c7" (UID: "54a80c44-b6f8-44bc-9a7d-86a43f48f7c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:28.723496 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.723330    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-workload-certs" pod "54a80c44-b6f8-44bc-9a7d-86a43f48f7c7" (UID: "54a80c44-b6f8-44bc-9a7d-86a43f48f7c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:28.723496 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.723342    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istiod-ca-cert" pod "54a80c44-b6f8-44bc-9a7d-86a43f48f7c7" (UID: "54a80c44-b6f8-44bc-9a7d-86a43f48f7c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:28.723632 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.723603    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-credential-socket" pod "54a80c44-b6f8-44bc-9a7d-86a43f48f7c7" (UID: "54a80c44-b6f8-44bc-9a7d-86a43f48f7c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:28.724033 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.724008    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:28.724124 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.724041    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:28.724124 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.724056    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:28.724124 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.724072    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:28.724124 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.724085    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:28.725225 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.725197    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-token" pod "54a80c44-b6f8-44bc-9a7d-86a43f48f7c7" (UID: "54a80c44-b6f8-44bc-9a7d-86a43f48f7c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:28.725365 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.725349    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-podinfo" pod "54a80c44-b6f8-44bc-9a7d-86a43f48f7c7" (UID: "54a80c44-b6f8-44bc-9a7d-86a43f48f7c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:20:28.725771 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.725746    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-envoy" pod "54a80c44-b6f8-44bc-9a7d-86a43f48f7c7" (UID: "54a80c44-b6f8-44bc-9a7d-86a43f48f7c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:28.725852 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.725809    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-kube-api-access-q9ss5" pod "54a80c44-b6f8-44bc-9a7d-86a43f48f7c7" (UID: "54a80c44-b6f8-44bc-9a7d-86a43f48f7c7"). InnerVolumeSpecName "kube-api-access-q9ss5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:28.825334 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.825307    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:28.825334 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.825331    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9ss5\" (UniqueName: \"kubernetes.io/projected/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-kube-api-access-q9ss5\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:28.825496 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.825342    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:28.825496 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:28.825351    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:29.419848 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.419824    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:20:29.534484 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.534403    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bfda83c-7138-4974-ae1c-67fa552ac430-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bfda83c-7138-4974-ae1c-67fa552ac430-istiod-ca-cert\") pod \"8bfda83c-7138-4974-ae1c-67fa552ac430\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") "
Oct 30 20:20:29.534484 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.534436    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-podinfo\") pod \"8bfda83c-7138-4974-ae1c-67fa552ac430\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") "
Oct 30 20:20:29.534638 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.534484    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-credential-socket\") pod \"8bfda83c-7138-4974-ae1c-67fa552ac430\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") "
Oct 30 20:20:29.534638 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.534525    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-workload-socket\") pod \"8bfda83c-7138-4974-ae1c-67fa552ac430\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") "
Oct 30 20:20:29.534638 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.534602    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-envoy\") pod \"8bfda83c-7138-4974-ae1c-67fa552ac430\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") "
Oct 30 20:20:29.534788 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.534665    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-workload-certs\") pod \"8bfda83c-7138-4974-ae1c-67fa552ac430\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") "
Oct 30 20:20:29.534841 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.534787    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-credential-socket" pod "8bfda83c-7138-4974-ae1c-67fa552ac430" (UID: "8bfda83c-7138-4974-ae1c-67fa552ac430"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:29.534841 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.534814    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-token\" (UniqueName: \"kubernetes.io/projected/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-token\") pod \"8bfda83c-7138-4974-ae1c-67fa552ac430\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") "
Oct 30 20:20:29.534945 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.534791    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bfda83c-7138-4974-ae1c-67fa552ac430-istiod-ca-cert" pod "8bfda83c-7138-4974-ae1c-67fa552ac430" (UID: "8bfda83c-7138-4974-ae1c-67fa552ac430"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:29.534945 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.534884    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-workload-socket" pod "8bfda83c-7138-4974-ae1c-67fa552ac430" (UID: "8bfda83c-7138-4974-ae1c-67fa552ac430"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:29.534945 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.534936    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bfda83c-7138-4974-ae1c-67fa552ac430-kube-api-access-rrm76\" (UniqueName: \"kubernetes.io/projected/8bfda83c-7138-4974-ae1c-67fa552ac430-kube-api-access-rrm76\") pod \"8bfda83c-7138-4974-ae1c-67fa552ac430\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") "
Oct 30 20:20:29.535094 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.534982    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-data\") pod \"8bfda83c-7138-4974-ae1c-67fa552ac430\" (UID: \"8bfda83c-7138-4974-ae1c-67fa552ac430\") "
Oct 30 20:20:29.535192 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.535168    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-workload-certs" pod "8bfda83c-7138-4974-ae1c-67fa552ac430" (UID: "8bfda83c-7138-4974-ae1c-67fa552ac430"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:29.535404 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.535383    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-data" pod "8bfda83c-7138-4974-ae1c-67fa552ac430" (UID: "8bfda83c-7138-4974-ae1c-67fa552ac430"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:29.535787 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.535763    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:29.535881 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.535794    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:29.535881 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.535808    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bfda83c-7138-4974-ae1c-67fa552ac430-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:29.535881 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.535825    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:29.535881 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.535838    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:29.536921 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.536898    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-envoy" pod "8bfda83c-7138-4974-ae1c-67fa552ac430" (UID: "8bfda83c-7138-4974-ae1c-67fa552ac430"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:29.537030 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.536900    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-podinfo" pod "8bfda83c-7138-4974-ae1c-67fa552ac430" (UID: "8bfda83c-7138-4974-ae1c-67fa552ac430"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:20:29.537158 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.537144    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-token" pod "8bfda83c-7138-4974-ae1c-67fa552ac430" (UID: "8bfda83c-7138-4974-ae1c-67fa552ac430"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:29.537504 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.537489    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bfda83c-7138-4974-ae1c-67fa552ac430-kube-api-access-rrm76" pod "8bfda83c-7138-4974-ae1c-67fa552ac430" (UID: "8bfda83c-7138-4974-ae1c-67fa552ac430"). InnerVolumeSpecName "kube-api-access-rrm76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:29.637783 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.637749    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:29.637783 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.637787    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrm76\" (UniqueName: \"kubernetes.io/projected/8bfda83c-7138-4974-ae1c-67fa552ac430-kube-api-access-rrm76\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:29.637961 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.637802    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:29.637961 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.637816    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bfda83c-7138-4974-ae1c-67fa552ac430-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:29.652856 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.652818    5958 generic.go:356] "Generic (PLEG): container finished" podID="8bfda83c-7138-4974-ae1c-67fa552ac430" containerID="fbea68500be3cc29b5255741f5efb36efa28803ca2f10d75c30ac94f9bfd9455" exitCode=0
Oct 30 20:20:29.653289 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.652874    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"
Oct 30 20:20:29.653289 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.652942    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675" event={"ID":"8bfda83c-7138-4974-ae1c-67fa552ac430","Type":"ContainerDied","Data":"fbea68500be3cc29b5255741f5efb36efa28803ca2f10d75c30ac94f9bfd9455"}
Oct 30 20:20:29.653289 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.652991    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675" event={"ID":"8bfda83c-7138-4974-ae1c-67fa552ac430","Type":"ContainerDied","Data":"58540b4440d393abac019b6b33c4bfd2b5947000cfcb7b979be82d19d253fdb2"}
Oct 30 20:20:29.653289 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.653022    5958 scope.go:122] "RemoveContainer" containerID="fbea68500be3cc29b5255741f5efb36efa28803ca2f10d75c30ac94f9bfd9455"
Oct 30 20:20:29.672627 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.672591    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv" event={"ID":"54a80c44-b6f8-44bc-9a7d-86a43f48f7c7","Type":"ContainerDied","Data":"28085f9cd55b62c34f3c25240673699b9b114053bd3c8e9ab75d96731eb17a54"}
Oct 30 20:20:29.672768 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.672608    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"
Oct 30 20:20:29.684755 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.684736    5958 scope.go:122] "RemoveContainer" containerID="fbea68500be3cc29b5255741f5efb36efa28803ca2f10d75c30ac94f9bfd9455"
Oct 30 20:20:29.685041 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:20:29.685021    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbea68500be3cc29b5255741f5efb36efa28803ca2f10d75c30ac94f9bfd9455\": container with ID starting with fbea68500be3cc29b5255741f5efb36efa28803ca2f10d75c30ac94f9bfd9455 not found: ID does not exist" containerID="fbea68500be3cc29b5255741f5efb36efa28803ca2f10d75c30ac94f9bfd9455"
Oct 30 20:20:29.685106 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.685049    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbea68500be3cc29b5255741f5efb36efa28803ca2f10d75c30ac94f9bfd9455"} err="failed to get container status \"fbea68500be3cc29b5255741f5efb36efa28803ca2f10d75c30ac94f9bfd9455\": rpc error: code = NotFound desc = could not find container \"fbea68500be3cc29b5255741f5efb36efa28803ca2f10d75c30ac94f9bfd9455\": container with ID starting with fbea68500be3cc29b5255741f5efb36efa28803ca2f10d75c30ac94f9bfd9455 not found: ID does not exist"
Oct 30 20:20:29.685106 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.685066    5958 scope.go:122] "RemoveContainer" containerID="037fa331b2b7055496b4a412573cedb6bb8b548e78af4530d27cd1a203f583dd"
Oct 30 20:20:29.722135 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:20:29.722095    5958 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bfda83c_7138_4974_ae1c_67fa552ac430.slice/crio-58540b4440d393abac019b6b33c4bfd2b5947000cfcb7b979be82d19d253fdb2\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bfda83c_7138_4974_ae1c_67fa552ac430.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 20:20:29.829632 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.829594    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"]
Oct 30 20:20:29.835310 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.835279    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-7qjbv"]
Oct 30 20:20:29.963807 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.963760    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"]
Oct 30 20:20:29.967510 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:29.967475    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-w6675"]
Oct 30 20:20:31.401777 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:31.401741    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54a80c44-b6f8-44bc-9a7d-86a43f48f7c7" path="/var/lib/kubelet/pods/54a80c44-b6f8-44bc-9a7d-86a43f48f7c7/volumes"
Oct 30 20:20:31.402256 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:31.402240    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bfda83c-7138-4974-ae1c-67fa552ac430" path="/var/lib/kubelet/pods/8bfda83c-7138-4974-ae1c-67fa552ac430/volumes"
Oct 30 20:20:31.422320 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:31.422295    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd"]
Oct 30 20:20:31.571969 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:31.571940    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f"]
Oct 30 20:20:36.423611 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:36.423572    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd" podUID="3a3a634a-354d-44eb-a927-123c5afe109f" containerName="payload-processing" containerID="cri-o://d93dec3bb905d43029811adc6dfda80b4e11947e0a4e86cbfaab90d7590c0e63" gracePeriod=25
Oct 30 20:20:36.572801 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:36.572755    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f" podUID="b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a" containerName="payload-pre-processing" containerID="cri-o://06097316ee435416be1fcf581464f1177945d442c7bc9034fab7b12a55ff5bc7" gracePeriod=25
Oct 30 20:20:36.616415 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:20:36.615707    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:20:36.698972 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:36.698944    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd"
Oct 30 20:20:36.731420 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:36.731391    5958 generic.go:356] "Generic (PLEG): container finished" podID="b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a" containerID="06097316ee435416be1fcf581464f1177945d442c7bc9034fab7b12a55ff5bc7" exitCode=0
Oct 30 20:20:36.731594 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:36.731469    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f" event={"ID":"b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a","Type":"ContainerDied","Data":"06097316ee435416be1fcf581464f1177945d442c7bc9034fab7b12a55ff5bc7"}
Oct 30 20:20:36.732972 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:36.732948    5958 generic.go:356] "Generic (PLEG): container finished" podID="3a3a634a-354d-44eb-a927-123c5afe109f" containerID="d93dec3bb905d43029811adc6dfda80b4e11947e0a4e86cbfaab90d7590c0e63" exitCode=0
Oct 30 20:20:36.733097 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:36.733010    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd"
Oct 30 20:20:36.733097 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:36.733027    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd" event={"ID":"3a3a634a-354d-44eb-a927-123c5afe109f","Type":"ContainerDied","Data":"d93dec3bb905d43029811adc6dfda80b4e11947e0a4e86cbfaab90d7590c0e63"}
Oct 30 20:20:36.733181 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:36.733111    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd" event={"ID":"3a3a634a-354d-44eb-a927-123c5afe109f","Type":"ContainerDied","Data":"7dfe3e40a195561d3a457917333ff1b11f3d152788d0abb86426bcf9d7e02e2e"}
Oct 30 20:20:36.733181 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:36.733140    5958 scope.go:122] "RemoveContainer" containerID="d93dec3bb905d43029811adc6dfda80b4e11947e0a4e86cbfaab90d7590c0e63"
Oct 30 20:20:36.746102 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:36.743686    5958 scope.go:122] "RemoveContainer" containerID="d93dec3bb905d43029811adc6dfda80b4e11947e0a4e86cbfaab90d7590c0e63"
Oct 30 20:20:36.746102 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:20:36.744016    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d93dec3bb905d43029811adc6dfda80b4e11947e0a4e86cbfaab90d7590c0e63\": container with ID starting with d93dec3bb905d43029811adc6dfda80b4e11947e0a4e86cbfaab90d7590c0e63 not found: ID does not exist" containerID="d93dec3bb905d43029811adc6dfda80b4e11947e0a4e86cbfaab90d7590c0e63"
Oct 30 20:20:36.746102 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:36.744042    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d93dec3bb905d43029811adc6dfda80b4e11947e0a4e86cbfaab90d7590c0e63"} err="failed to get container status \"d93dec3bb905d43029811adc6dfda80b4e11947e0a4e86cbfaab90d7590c0e63\": rpc error: code = NotFound desc = could not find container \"d93dec3bb905d43029811adc6dfda80b4e11947e0a4e86cbfaab90d7590c0e63\": container with ID starting with d93dec3bb905d43029811adc6dfda80b4e11947e0a4e86cbfaab90d7590c0e63 not found: ID does not exist"
Oct 30 20:20:36.804254 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:36.804212    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a3a634a-354d-44eb-a927-123c5afe109f-kube-api-access-n8ptb\" (UniqueName: \"kubernetes.io/projected/3a3a634a-354d-44eb-a927-123c5afe109f-kube-api-access-n8ptb\") pod \"3a3a634a-354d-44eb-a927-123c5afe109f\" (UID: \"3a3a634a-354d-44eb-a927-123c5afe109f\") "
Oct 30 20:20:36.804553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:36.804529    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a3a634a-354d-44eb-a927-123c5afe109f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3a634a-354d-44eb-a927-123c5afe109f-plugins-config-volume\") pod \"3a3a634a-354d-44eb-a927-123c5afe109f\" (UID: \"3a3a634a-354d-44eb-a927-123c5afe109f\") "
Oct 30 20:20:36.805172 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:36.805123    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a3a634a-354d-44eb-a927-123c5afe109f-plugins-config-volume" pod "3a3a634a-354d-44eb-a927-123c5afe109f" (UID: "3a3a634a-354d-44eb-a927-123c5afe109f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:36.806557 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:36.806533    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a3a634a-354d-44eb-a927-123c5afe109f-kube-api-access-n8ptb" pod "3a3a634a-354d-44eb-a927-123c5afe109f" (UID: "3a3a634a-354d-44eb-a927-123c5afe109f"). InnerVolumeSpecName "kube-api-access-n8ptb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:36.832023 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:36.831995    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f"
Oct 30 20:20:36.905880 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:36.905851    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a-kube-api-access-7qck4\" (UniqueName: \"kubernetes.io/projected/b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a-kube-api-access-7qck4\") pod \"b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a\" (UID: \"b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a\") "
Oct 30 20:20:36.906023 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:36.905974    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a-plugins-config-volume\") pod \"b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a\" (UID: \"b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a\") "
Oct 30 20:20:36.906294 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:36.906271    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a-plugins-config-volume" pod "b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a" (UID: "b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:36.906678 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:36.906661    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:36.906731 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:36.906681    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a3a634a-354d-44eb-a927-123c5afe109f-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:36.906731 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:36.906691    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8ptb\" (UniqueName: \"kubernetes.io/projected/3a3a634a-354d-44eb-a927-123c5afe109f-kube-api-access-n8ptb\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:36.908003 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:36.907983    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a-kube-api-access-7qck4" pod "b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a" (UID: "b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a"). InnerVolumeSpecName "kube-api-access-7qck4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:37.007698 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:37.007625    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qck4\" (UniqueName: \"kubernetes.io/projected/b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a-kube-api-access-7qck4\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:37.159171 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:37.159134    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd"]
Oct 30 20:20:37.161798 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:37.161773    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a2103a-7dfcf67b4b-xwdnd"]
Oct 30 20:20:37.404858 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:37.404815    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a3a634a-354d-44eb-a927-123c5afe109f" path="/var/lib/kubelet/pods/3a3a634a-354d-44eb-a927-123c5afe109f/volumes"
Oct 30 20:20:37.741292 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:37.741211    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f"
Oct 30 20:20:37.741292 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:37.741245    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f" event={"ID":"b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a","Type":"ContainerDied","Data":"728458105aedebb514126bb5013caa203ca6d307e024b665dd7c26bb8213eb9b"}
Oct 30 20:20:37.741838 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:37.741312    5958 scope.go:122] "RemoveContainer" containerID="06097316ee435416be1fcf581464f1177945d442c7bc9034fab7b12a55ff5bc7"
Oct 30 20:20:37.910585 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:37.910553    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f"]
Oct 30 20:20:37.960388 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:37.960362    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a2103a-bf7d54fbc-zjl5f"]
Oct 30 20:20:39.399379 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:39.399336    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a" path="/var/lib/kubelet/pods/b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a/volumes"
Oct 30 20:20:49.971739 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:49.971702    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"]
Oct 30 20:20:49.972157 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:49.971899    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc" podUID="9f8392a2-9219-4e55-88a8-955e72c0f98c" containerName="discovery" containerID="cri-o://4295f80209bbd28b13aed32dfd2c9e4e1b045c65b8fc369f964f5c71c5b87fa6" gracePeriod=30
Oct 30 20:20:50.248727 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.248687    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:20:50.431236 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.431199    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8392a2-9219-4e55-88a8-955e72c0f98c-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8392a2-9219-4e55-88a8-955e72c0f98c-local-certs\") pod \"9f8392a2-9219-4e55-88a8-955e72c0f98c\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") "
Oct 30 20:20:50.431540 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.431508    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-csr-dns-cert\") pod \"9f8392a2-9219-4e55-88a8-955e72c0f98c\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") "
Oct 30 20:20:50.431684 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.431567    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f8392a2-9219-4e55-88a8-955e72c0f98c-kube-api-access-rnjcn\" (UniqueName: \"kubernetes.io/projected/9f8392a2-9219-4e55-88a8-955e72c0f98c-kube-api-access-rnjcn\") pod \"9f8392a2-9219-4e55-88a8-955e72c0f98c\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") "
Oct 30 20:20:50.431757 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.431711    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-token\" (UniqueName: \"kubernetes.io/projected/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-token\") pod \"9f8392a2-9219-4e55-88a8-955e72c0f98c\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") "
Oct 30 20:20:50.431834 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.431798    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f8392a2-9219-4e55-88a8-955e72c0f98c-cacerts\" (UniqueName: \"kubernetes.io/secret/9f8392a2-9219-4e55-88a8-955e72c0f98c-cacerts\") pod \"9f8392a2-9219-4e55-88a8-955e72c0f98c\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") "
Oct 30 20:20:50.431938 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.431923    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-kubeconfig\") pod \"9f8392a2-9219-4e55-88a8-955e72c0f98c\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") "
Oct 30 20:20:50.433348 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.432031    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-csr-ca-configmap\") pod \"9f8392a2-9219-4e55-88a8-955e72c0f98c\" (UID: \"9f8392a2-9219-4e55-88a8-955e72c0f98c\") "
Oct 30 20:20:50.433348 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.433065    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-csr-ca-configmap" pod "9f8392a2-9219-4e55-88a8-955e72c0f98c" (UID: "9f8392a2-9219-4e55-88a8-955e72c0f98c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:50.433348 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.433291    5958 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-csr-ca-configmap\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:50.442941 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.442865    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-kubeconfig" pod "9f8392a2-9219-4e55-88a8-955e72c0f98c" (UID: "9f8392a2-9219-4e55-88a8-955e72c0f98c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:50.442941 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.442888    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-csr-dns-cert" pod "9f8392a2-9219-4e55-88a8-955e72c0f98c" (UID: "9f8392a2-9219-4e55-88a8-955e72c0f98c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:50.443225 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.442965    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f8392a2-9219-4e55-88a8-955e72c0f98c-cacerts" pod "9f8392a2-9219-4e55-88a8-955e72c0f98c" (UID: "9f8392a2-9219-4e55-88a8-955e72c0f98c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:50.443554 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.443520    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-token" pod "9f8392a2-9219-4e55-88a8-955e72c0f98c" (UID: "9f8392a2-9219-4e55-88a8-955e72c0f98c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:50.443667 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.443588    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f8392a2-9219-4e55-88a8-955e72c0f98c-kube-api-access-rnjcn" pod "9f8392a2-9219-4e55-88a8-955e72c0f98c" (UID: "9f8392a2-9219-4e55-88a8-955e72c0f98c"). InnerVolumeSpecName "kube-api-access-rnjcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:50.446240 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.446203    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8392a2-9219-4e55-88a8-955e72c0f98c-local-certs" pod "9f8392a2-9219-4e55-88a8-955e72c0f98c" (UID: "9f8392a2-9219-4e55-88a8-955e72c0f98c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:20:50.535019 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.534936    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:50.535169 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.535033    5958 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f8392a2-9219-4e55-88a8-955e72c0f98c-cacerts\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:50.535169 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.535053    5958 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-kubeconfig\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:50.535169 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.535063    5958 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8392a2-9219-4e55-88a8-955e72c0f98c-local-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:50.535169 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.535073    5958 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f8392a2-9219-4e55-88a8-955e72c0f98c-istio-csr-dns-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:50.535169 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.535081    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnjcn\" (UniqueName: \"kubernetes.io/projected/9f8392a2-9219-4e55-88a8-955e72c0f98c-kube-api-access-rnjcn\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:50.858809 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.858776    5958 generic.go:356] "Generic (PLEG): container finished" podID="9f8392a2-9219-4e55-88a8-955e72c0f98c" containerID="4295f80209bbd28b13aed32dfd2c9e4e1b045c65b8fc369f964f5c71c5b87fa6" exitCode=0
Oct 30 20:20:50.859006 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.858821    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"
Oct 30 20:20:50.859006 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.858866    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc" event={"ID":"9f8392a2-9219-4e55-88a8-955e72c0f98c","Type":"ContainerDied","Data":"4295f80209bbd28b13aed32dfd2c9e4e1b045c65b8fc369f964f5c71c5b87fa6"}
Oct 30 20:20:50.859006 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.858915    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc" event={"ID":"9f8392a2-9219-4e55-88a8-955e72c0f98c","Type":"ContainerDied","Data":"8ea7b8252c165b6c2b511b12e03f4c751133b915775caec49fd66d1dd34b8051"}
Oct 30 20:20:50.859006 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.858935    5958 scope.go:122] "RemoveContainer" containerID="4295f80209bbd28b13aed32dfd2c9e4e1b045c65b8fc369f964f5c71c5b87fa6"
Oct 30 20:20:50.869710 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.869687    5958 scope.go:122] "RemoveContainer" containerID="4295f80209bbd28b13aed32dfd2c9e4e1b045c65b8fc369f964f5c71c5b87fa6"
Oct 30 20:20:50.870010 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:20:50.869989    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4295f80209bbd28b13aed32dfd2c9e4e1b045c65b8fc369f964f5c71c5b87fa6\": container with ID starting with 4295f80209bbd28b13aed32dfd2c9e4e1b045c65b8fc369f964f5c71c5b87fa6 not found: ID does not exist" containerID="4295f80209bbd28b13aed32dfd2c9e4e1b045c65b8fc369f964f5c71c5b87fa6"
Oct 30 20:20:50.870061 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:50.870019    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4295f80209bbd28b13aed32dfd2c9e4e1b045c65b8fc369f964f5c71c5b87fa6"} err="failed to get container status \"4295f80209bbd28b13aed32dfd2c9e4e1b045c65b8fc369f964f5c71c5b87fa6\": rpc error: code = NotFound desc = could not find container \"4295f80209bbd28b13aed32dfd2c9e4e1b045c65b8fc369f964f5c71c5b87fa6\": container with ID starting with 4295f80209bbd28b13aed32dfd2c9e4e1b045c65b8fc369f964f5c71c5b87fa6 not found: ID does not exist"
Oct 30 20:20:51.025736 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:51.025696    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"]
Oct 30 20:20:51.075359 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:51.075315    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bkjvc"]
Oct 30 20:20:51.399181 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:51.399145    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f8392a2-9219-4e55-88a8-955e72c0f98c" path="/var/lib/kubelet/pods/9f8392a2-9219-4e55-88a8-955e72c0f98c/volumes"
Oct 30 20:20:58.599972 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.599935    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d"]
Oct 30 20:20:58.600625 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600606    5958 cpu_manager.go:414] "containerMap: removing container" podUID="23b32875-225b-469d-8794-c47053a71ebc" containerName="istio-proxy"
Oct 30 20:20:58.600690 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600628    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23b32875-225b-469d-8794-c47053a71ebc" containerName="istio-proxy"
Oct 30 20:20:58.600690 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600642    5958 cpu_manager.go:414] "containerMap: removing container" podUID="54a80c44-b6f8-44bc-9a7d-86a43f48f7c7" containerName="istio-proxy"
Oct 30 20:20:58.600690 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600647    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54a80c44-b6f8-44bc-9a7d-86a43f48f7c7" containerName="istio-proxy"
Oct 30 20:20:58.600690 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600656    5958 cpu_manager.go:414] "containerMap: removing container" podUID="a820cb88-eb76-459c-bb08-28837aa714a1" containerName="cleanup"
Oct 30 20:20:58.600690 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600664    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a820cb88-eb76-459c-bb08-28837aa714a1" containerName="cleanup"
Oct 30 20:20:58.600690 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600675    5958 cpu_manager.go:414] "containerMap: removing container" podUID="23b32875-225b-469d-8794-c47053a71ebc" containerName="istio-proxy"
Oct 30 20:20:58.600690 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600679    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23b32875-225b-469d-8794-c47053a71ebc" containerName="istio-proxy"
Oct 30 20:20:58.600904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600692    5958 cpu_manager.go:414] "containerMap: removing container" podUID="be48f946-80c5-464d-bd18-408c6d239046" containerName="payload-pre-processing"
Oct 30 20:20:58.600904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600700    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be48f946-80c5-464d-bd18-408c6d239046" containerName="payload-pre-processing"
Oct 30 20:20:58.600904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600710    5958 cpu_manager.go:414] "containerMap: removing container" podUID="9f8392a2-9219-4e55-88a8-955e72c0f98c" containerName="discovery"
Oct 30 20:20:58.600904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600716    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f8392a2-9219-4e55-88a8-955e72c0f98c" containerName="discovery"
Oct 30 20:20:58.600904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600726    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3a3a634a-354d-44eb-a927-123c5afe109f" containerName="payload-processing"
Oct 30 20:20:58.600904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600731    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a3a634a-354d-44eb-a927-123c5afe109f" containerName="payload-processing"
Oct 30 20:20:58.600904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600737    5958 cpu_manager.go:414] "containerMap: removing container" podUID="fc5eb38c-c314-4a32-846f-d5b63f2ad009" containerName="payload-processing"
Oct 30 20:20:58.600904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600741    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc5eb38c-c314-4a32-846f-d5b63f2ad009" containerName="payload-processing"
Oct 30 20:20:58.600904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600752    5958 cpu_manager.go:414] "containerMap: removing container" podUID="7c2eca7f-c86c-427e-b1c5-1d33ded45e42" containerName="istio-proxy"
Oct 30 20:20:58.600904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600756    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c2eca7f-c86c-427e-b1c5-1d33ded45e42" containerName="istio-proxy"
Oct 30 20:20:58.600904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600765    5958 cpu_manager.go:414] "containerMap: removing container" podUID="a1551c50-4d89-4227-838c-4644c41de49b" containerName="main"
Oct 30 20:20:58.600904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600771    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1551c50-4d89-4227-838c-4644c41de49b" containerName="main"
Oct 30 20:20:58.600904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600786    5958 cpu_manager.go:414] "containerMap: removing container" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main"
Oct 30 20:20:58.600904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600791    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main"
Oct 30 20:20:58.600904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600809    5958 cpu_manager.go:414] "containerMap: removing container" podUID="8bfda83c-7138-4974-ae1c-67fa552ac430" containerName="istio-proxy"
Oct 30 20:20:58.600904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600813    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bfda83c-7138-4974-ae1c-67fa552ac430" containerName="istio-proxy"
Oct 30 20:20:58.600904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600820    5958 cpu_manager.go:414] "containerMap: removing container" podUID="b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a" containerName="payload-pre-processing"
Oct 30 20:20:58.600904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600825    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a" containerName="payload-pre-processing"
Oct 30 20:20:58.600904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600832    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ed65872e-55ca-416c-bf1d-05e1bc0047b5" containerName="cleanup"
Oct 30 20:20:58.600904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600836    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed65872e-55ca-416c-bf1d-05e1bc0047b5" containerName="cleanup"
Oct 30 20:20:58.600904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600850    5958 cpu_manager.go:414] "containerMap: removing container" podUID="b8b98921-b374-47ca-93d1-6ccd66cae325" containerName="maas-api"
Oct 30 20:20:58.600904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600857    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b98921-b374-47ca-93d1-6ccd66cae325" containerName="maas-api"
Oct 30 20:20:58.601573 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600941    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7fae60d-346d-4d4a-8851-82586d2ddf65" containerName="cleanup"
Oct 30 20:20:58.601573 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600950    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2103c1-53ee-4aaf-ab8e-75c5c93390a1" containerName="cleanup"
Oct 30 20:20:58.601573 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600957    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bfda83c-7138-4974-ae1c-67fa552ac430" containerName="istio-proxy"
Oct 30 20:20:58.601573 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600964    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="b60fd1ec-0b05-41d3-a9ba-8d1ed3e5904a" containerName="payload-pre-processing"
Oct 30 20:20:58.601573 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600972    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b98921-b374-47ca-93d1-6ccd66cae325" containerName="maas-api"
Oct 30 20:20:58.601573 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600978    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="23b32875-225b-469d-8794-c47053a71ebc" containerName="istio-proxy"
Oct 30 20:20:58.601573 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600984    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="54a80c44-b6f8-44bc-9a7d-86a43f48f7c7" containerName="istio-proxy"
Oct 30 20:20:58.601573 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600991    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="23b32875-225b-469d-8794-c47053a71ebc" containerName="istio-proxy"
Oct 30 20:20:58.601573 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.600999    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="be48f946-80c5-464d-bd18-408c6d239046" containerName="payload-pre-processing"
Oct 30 20:20:58.601573 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.601009    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f8392a2-9219-4e55-88a8-955e72c0f98c" containerName="discovery"
Oct 30 20:20:58.601573 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.601019    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a3a634a-354d-44eb-a927-123c5afe109f" containerName="payload-processing"
Oct 30 20:20:58.601573 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.601024    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc5eb38c-c314-4a32-846f-d5b63f2ad009" containerName="payload-processing"
Oct 30 20:20:58.601573 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.601031    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c2eca7f-c86c-427e-b1c5-1d33ded45e42" containerName="istio-proxy"
Oct 30 20:20:58.601573 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.601037    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1551c50-4d89-4227-838c-4644c41de49b" containerName="main"
Oct 30 20:20:58.601573 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.601043    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1929aa5-fcd4-4609-8cbc-68afb34bdadc" containerName="main"
Oct 30 20:20:58.606106 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.606087    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d"
Oct 30 20:20:58.661984 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.661949    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d"]
Oct 30 20:20:58.717968 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.717931    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-01e8dc\""
Oct 30 20:20:58.718642 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.718475    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-01e8dc-dockercfg-5489q\""
Oct 30 20:20:58.730680 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.730653    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7t4c\" (UniqueName: \"kubernetes.io/projected/3224f5ec-0f50-454e-a07d-e16c0b9fe71b-kube-api-access-r7t4c\") pod \"payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d\" (UID: \"3224f5ec-0f50-454e-a07d-e16c0b9fe71b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d"
Oct 30 20:20:58.730905 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.730880    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3224f5ec-0f50-454e-a07d-e16c0b9fe71b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d\" (UID: \"3224f5ec-0f50-454e-a07d-e16c0b9fe71b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d"
Oct 30 20:20:58.761227 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.761194    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc"]
Oct 30 20:20:58.765849 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.765824    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc"
Oct 30 20:20:58.818796 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.818760    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc"]
Oct 30 20:20:58.833306 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.833265    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5ssn\" (UniqueName: \"kubernetes.io/projected/7578c703-e969-495b-98e9-70bfdf1f8413-kube-api-access-b5ssn\") pod \"payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc\" (UID: \"7578c703-e969-495b-98e9-70bfdf1f8413\") " pod="openshift-ingress/payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc"
Oct 30 20:20:58.833829 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.833785    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7578c703-e969-495b-98e9-70bfdf1f8413-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc\" (UID: \"7578c703-e969-495b-98e9-70bfdf1f8413\") " pod="openshift-ingress/payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc"
Oct 30 20:20:58.833931 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.833877    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7t4c\" (UniqueName: \"kubernetes.io/projected/3224f5ec-0f50-454e-a07d-e16c0b9fe71b-kube-api-access-r7t4c\") pod \"payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d\" (UID: \"3224f5ec-0f50-454e-a07d-e16c0b9fe71b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d"
Oct 30 20:20:58.833997 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.833928    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3224f5ec-0f50-454e-a07d-e16c0b9fe71b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d\" (UID: \"3224f5ec-0f50-454e-a07d-e16c0b9fe71b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d"
Oct 30 20:20:58.835188 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.835035    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3224f5ec-0f50-454e-a07d-e16c0b9fe71b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d\" (UID: \"3224f5ec-0f50-454e-a07d-e16c0b9fe71b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d"
Oct 30 20:20:58.896759 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.896690    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7t4c\" (UniqueName: \"kubernetes.io/projected/3224f5ec-0f50-454e-a07d-e16c0b9fe71b-kube-api-access-r7t4c\") pod \"payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d\" (UID: \"3224f5ec-0f50-454e-a07d-e16c0b9fe71b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d"
Oct 30 20:20:58.919903 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.919875    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d"
Oct 30 20:20:58.935662 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.935634    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7578c703-e969-495b-98e9-70bfdf1f8413-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc\" (UID: \"7578c703-e969-495b-98e9-70bfdf1f8413\") " pod="openshift-ingress/payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc"
Oct 30 20:20:58.935838 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.935818    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5ssn\" (UniqueName: \"kubernetes.io/projected/7578c703-e969-495b-98e9-70bfdf1f8413-kube-api-access-b5ssn\") pod \"payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc\" (UID: \"7578c703-e969-495b-98e9-70bfdf1f8413\") " pod="openshift-ingress/payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc"
Oct 30 20:20:58.936231 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:58.936211    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7578c703-e969-495b-98e9-70bfdf1f8413-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc\" (UID: \"7578c703-e969-495b-98e9-70bfdf1f8413\") " pod="openshift-ingress/payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc"
Oct 30 20:20:59.002132 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:59.002101    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5ssn\" (UniqueName: \"kubernetes.io/projected/7578c703-e969-495b-98e9-70bfdf1f8413-kube-api-access-b5ssn\") pod \"payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc\" (UID: \"7578c703-e969-495b-98e9-70bfdf1f8413\") " pod="openshift-ingress/payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc"
Oct 30 20:20:59.081494 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:59.081435    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc"
Oct 30 20:20:59.299975 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:59.299942    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d"]
Oct 30 20:20:59.301688 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:20:59.301429    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3224f5ec_0f50_454e_a07d_e16c0b9fe71b.slice/crio-45fd528a39e6b845dec863efb19bdcb0495ab961505e112502b5feee26af40d0 WatchSource:0}: Error finding container 45fd528a39e6b845dec863efb19bdcb0495ab961505e112502b5feee26af40d0: Status 404 returned error can't find the container with id 45fd528a39e6b845dec863efb19bdcb0495ab961505e112502b5feee26af40d0
Oct 30 20:20:59.314086 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:59.314054    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5623f05-snp8l"]
Oct 30 20:20:59.320092 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:59.320067    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5623f05-snp8l"
Oct 30 20:20:59.375755 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:59.375660    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5623f05-snp8l"]
Oct 30 20:20:59.412502 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:59.412139    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:20:59.412502 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:59.412198    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:20:59.444962 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:59.444927    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmlqr\" (UniqueName: \"kubernetes.io/projected/faafca8a-42dc-4613-89fe-c135d4a7acab-kube-api-access-nmlqr\") pod \"maas-api-revoke-sub-b5623f05-snp8l\" (UID: \"faafca8a-42dc-4613-89fe-c135d4a7acab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5623f05-snp8l"
Oct 30 20:20:59.445166 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:59.445149    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faafca8a-42dc-4613-89fe-c135d4a7acab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5623f05-snp8l\" (UID: \"faafca8a-42dc-4613-89fe-c135d4a7acab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5623f05-snp8l"
Oct 30 20:20:59.445237 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:59.445210    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faafca8a-42dc-4613-89fe-c135d4a7acab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5623f05-snp8l\" (UID: \"faafca8a-42dc-4613-89fe-c135d4a7acab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5623f05-snp8l"
Oct 30 20:20:59.460364 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:59.460338    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc"]
Oct 30 20:20:59.461699 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:20:59.461668    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7578c703_e969_495b_98e9_70bfdf1f8413.slice/crio-564ce05c3615bf8239e07b7f1a0aae2a7d4ce74e42404ac7a5350a7c2aad4004 WatchSource:0}: Error finding container 564ce05c3615bf8239e07b7f1a0aae2a7d4ce74e42404ac7a5350a7c2aad4004: Status 404 returned error can't find the container with id 564ce05c3615bf8239e07b7f1a0aae2a7d4ce74e42404ac7a5350a7c2aad4004
Oct 30 20:20:59.546146 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:59.546118    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmlqr\" (UniqueName: \"kubernetes.io/projected/faafca8a-42dc-4613-89fe-c135d4a7acab-kube-api-access-nmlqr\") pod \"maas-api-revoke-sub-b5623f05-snp8l\" (UID: \"faafca8a-42dc-4613-89fe-c135d4a7acab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5623f05-snp8l"
Oct 30 20:20:59.546313 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:59.546297    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faafca8a-42dc-4613-89fe-c135d4a7acab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5623f05-snp8l\" (UID: \"faafca8a-42dc-4613-89fe-c135d4a7acab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5623f05-snp8l"
Oct 30 20:20:59.546469 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:59.546338    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faafca8a-42dc-4613-89fe-c135d4a7acab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5623f05-snp8l\" (UID: \"faafca8a-42dc-4613-89fe-c135d4a7acab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5623f05-snp8l"
Oct 30 20:20:59.546928 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:59.546906    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faafca8a-42dc-4613-89fe-c135d4a7acab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5623f05-snp8l\" (UID: \"faafca8a-42dc-4613-89fe-c135d4a7acab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5623f05-snp8l"
Oct 30 20:20:59.548631 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:59.548579    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faafca8a-42dc-4613-89fe-c135d4a7acab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5623f05-snp8l\" (UID: \"faafca8a-42dc-4613-89fe-c135d4a7acab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5623f05-snp8l"
Oct 30 20:20:59.608800 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:59.608767    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmlqr\" (UniqueName: \"kubernetes.io/projected/faafca8a-42dc-4613-89fe-c135d4a7acab-kube-api-access-nmlqr\") pod \"maas-api-revoke-sub-b5623f05-snp8l\" (UID: \"faafca8a-42dc-4613-89fe-c135d4a7acab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5623f05-snp8l"
Oct 30 20:20:59.657137 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:59.657106    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5623f05-snp8l"
Oct 30 20:20:59.926940 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:59.926901    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d" event={"ID":"3224f5ec-0f50-454e-a07d-e16c0b9fe71b","Type":"ContainerStarted","Data":"c60700ba8f7be5d773de166e95b86211d89ee969e1a7c6e58091f6c8860e9061"}
Oct 30 20:20:59.927161 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:59.926969    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d" event={"ID":"3224f5ec-0f50-454e-a07d-e16c0b9fe71b","Type":"ContainerStarted","Data":"45fd528a39e6b845dec863efb19bdcb0495ab961505e112502b5feee26af40d0"}
Oct 30 20:20:59.927307 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:59.927289    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d"
Oct 30 20:20:59.936271 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:59.936228    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc" event={"ID":"7578c703-e969-495b-98e9-70bfdf1f8413","Type":"ContainerStarted","Data":"4bb5fa5fc5771b0eaac80e7b62f4d2b00c0fe00ea18bbf0738dac21aa84d6122"}
Oct 30 20:20:59.936462 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:59.936303    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc" event={"ID":"7578c703-e969-495b-98e9-70bfdf1f8413","Type":"ContainerStarted","Data":"564ce05c3615bf8239e07b7f1a0aae2a7d4ce74e42404ac7a5350a7c2aad4004"}
Oct 30 20:20:59.936462 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:20:59.936330    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc"
Oct 30 20:21:00.037402 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:00.037370    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5623f05-snp8l"]
Oct 30 20:21:00.039096 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:21:00.039069    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaafca8a_42dc_4613_89fe_c135d4a7acab.slice/crio-17898b646d9cf33784da032ad8a1761e0e9ef31b4d866e5185b8eaeab5a7ee45 WatchSource:0}: Error finding container 17898b646d9cf33784da032ad8a1761e0e9ef31b4d866e5185b8eaeab5a7ee45: Status 404 returned error can't find the container with id 17898b646d9cf33784da032ad8a1761e0e9ef31b4d866e5185b8eaeab5a7ee45
Oct 30 20:21:00.046992 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:00.046951    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d" podStartSLOduration=2.046939357 podStartE2EDuration="2.046939357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:00.043130264 +0000 UTC m=+2298.775345656" watchObservedRunningTime="2026-10-30 20:21:00.046939357 +0000 UTC m=+2298.779154746"
Oct 30 20:21:00.163488 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:00.163417    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc" podStartSLOduration=2.163400342 podStartE2EDuration="2.163400342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:00.160585768 +0000 UTC m=+2298.892801157" watchObservedRunningTime="2026-10-30 20:21:00.163400342 +0000 UTC m=+2298.895615737"
Oct 30 20:21:00.945171 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:00.945134    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5623f05-snp8l" event={"ID":"faafca8a-42dc-4613-89fe-c135d4a7acab","Type":"ContainerStarted","Data":"cb1079cbfd7132da9b4c79351bde233b59a26ae23585c7fe9db9faf90199c5ea"}
Oct 30 20:21:00.945654 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:00.945197    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5623f05-snp8l" event={"ID":"faafca8a-42dc-4613-89fe-c135d4a7acab","Type":"ContainerStarted","Data":"17898b646d9cf33784da032ad8a1761e0e9ef31b4d866e5185b8eaeab5a7ee45"}
Oct 30 20:21:01.064690 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:01.064630    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5623f05-snp8l" podStartSLOduration=2.064616525 podStartE2EDuration="2.064616525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:01.061931675 +0000 UTC m=+2299.794147088" watchObservedRunningTime="2026-10-30 20:21:01.064616525 +0000 UTC m=+2299.796831911"
Oct 30 20:21:01.953651 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:01.953610    5958 generic.go:356] "Generic (PLEG): container finished" podID="faafca8a-42dc-4613-89fe-c135d4a7acab" containerID="cb1079cbfd7132da9b4c79351bde233b59a26ae23585c7fe9db9faf90199c5ea" exitCode=0
Oct 30 20:21:01.954044 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:01.953700    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5623f05-snp8l" event={"ID":"faafca8a-42dc-4613-89fe-c135d4a7acab","Type":"ContainerDied","Data":"cb1079cbfd7132da9b4c79351bde233b59a26ae23585c7fe9db9faf90199c5ea"}
Oct 30 20:21:03.119099 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:03.119073    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5623f05-snp8l"
Oct 30 20:21:03.195699 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:03.195668    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faafca8a-42dc-4613-89fe-c135d4a7acab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faafca8a-42dc-4613-89fe-c135d4a7acab-maas-api-openshift-ca\") pod \"faafca8a-42dc-4613-89fe-c135d4a7acab\" (UID: \"faafca8a-42dc-4613-89fe-c135d4a7acab\") "
Oct 30 20:21:03.195901 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:03.195731    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faafca8a-42dc-4613-89fe-c135d4a7acab-kube-api-access-nmlqr\" (UniqueName: \"kubernetes.io/projected/faafca8a-42dc-4613-89fe-c135d4a7acab-kube-api-access-nmlqr\") pod \"faafca8a-42dc-4613-89fe-c135d4a7acab\" (UID: \"faafca8a-42dc-4613-89fe-c135d4a7acab\") "
Oct 30 20:21:03.195901 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:03.195776    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faafca8a-42dc-4613-89fe-c135d4a7acab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faafca8a-42dc-4613-89fe-c135d4a7acab-maas-api-xks-ca\") pod \"faafca8a-42dc-4613-89fe-c135d4a7acab\" (UID: \"faafca8a-42dc-4613-89fe-c135d4a7acab\") "
Oct 30 20:21:03.196104 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:03.196080    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faafca8a-42dc-4613-89fe-c135d4a7acab-maas-api-openshift-ca" pod "faafca8a-42dc-4613-89fe-c135d4a7acab" (UID: "faafca8a-42dc-4613-89fe-c135d4a7acab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:03.196220 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:03.196205    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faafca8a-42dc-4613-89fe-c135d4a7acab-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:03.198085 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:03.198054    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faafca8a-42dc-4613-89fe-c135d4a7acab-maas-api-xks-ca" pod "faafca8a-42dc-4613-89fe-c135d4a7acab" (UID: "faafca8a-42dc-4613-89fe-c135d4a7acab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:03.198206 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:03.198133    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faafca8a-42dc-4613-89fe-c135d4a7acab-kube-api-access-nmlqr" pod "faafca8a-42dc-4613-89fe-c135d4a7acab" (UID: "faafca8a-42dc-4613-89fe-c135d4a7acab"). InnerVolumeSpecName "kube-api-access-nmlqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:03.297527 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:03.297403    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmlqr\" (UniqueName: \"kubernetes.io/projected/faafca8a-42dc-4613-89fe-c135d4a7acab-kube-api-access-nmlqr\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:03.297527 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:03.297446    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faafca8a-42dc-4613-89fe-c135d4a7acab-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:03.968177 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:03.968143    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5623f05-snp8l"
Oct 30 20:21:03.968376 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:03.968146    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5623f05-snp8l" event={"ID":"faafca8a-42dc-4613-89fe-c135d4a7acab","Type":"ContainerDied","Data":"17898b646d9cf33784da032ad8a1761e0e9ef31b4d866e5185b8eaeab5a7ee45"}
Oct 30 20:21:03.968376 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:03.968289    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17898b646d9cf33784da032ad8a1761e0e9ef31b4d866e5185b8eaeab5a7ee45"
Oct 30 20:21:06.624968 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:21:06.624926    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:21:10.961003 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:10.960972    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc"
Oct 30 20:21:10.961535 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:10.961511    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d"
Oct 30 20:21:12.981484 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:12.980707    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"]
Oct 30 20:21:12.995475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:12.992909    5958 cpu_manager.go:414] "containerMap: removing container" podUID="faafca8a-42dc-4613-89fe-c135d4a7acab" containerName="revoke-keys"
Oct 30 20:21:12.995475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:12.992940    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faafca8a-42dc-4613-89fe-c135d4a7acab" containerName="revoke-keys"
Oct 30 20:21:12.995475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:12.993086    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="faafca8a-42dc-4613-89fe-c135d4a7acab" containerName="revoke-keys"
Oct 30 20:21:13.031720 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.031691    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.072680 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.072640    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"]
Oct 30 20:21:13.118946 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.118895    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a2386c-openshift-default-dockercfg-rtttk\""
Oct 30 20:21:13.127236 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.124215    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-istio-data\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.127593 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.127565    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04b5051c-1bff-4550-b442-7f0b7622a271-istiod-ca-cert\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.127806 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.127785    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk6j6\" (UniqueName: \"kubernetes.io/projected/04b5051c-1bff-4550-b442-7f0b7622a271-kube-api-access-xk6j6\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.127901 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.127829    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04b5051c-1bff-4550-b442-7f0b7622a271-istio-token\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.127976 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.127960    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-istio-envoy\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.128139 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.128120    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04b5051c-1bff-4550-b442-7f0b7622a271-istio-podinfo\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.128238 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.128165    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-workload-socket\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.128238 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.128194    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-workload-certs\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.128348 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.128299    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-credential-socket\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.229266 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.229232    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-istio-data\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.229508 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.229325    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04b5051c-1bff-4550-b442-7f0b7622a271-istiod-ca-cert\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.229508 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.229412    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk6j6\" (UniqueName: \"kubernetes.io/projected/04b5051c-1bff-4550-b442-7f0b7622a271-kube-api-access-xk6j6\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.229508 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.229445    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04b5051c-1bff-4550-b442-7f0b7622a271-istio-token\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.229704 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.229602    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-istio-envoy\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.229704 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.229640    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04b5051c-1bff-4550-b442-7f0b7622a271-istio-podinfo\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.229704 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.229674    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-workload-socket\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.229704 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.229682    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-istio-data\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.229901 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.229704    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-workload-certs\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.229901 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.229771    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-credential-socket\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.230051 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.230027    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04b5051c-1bff-4550-b442-7f0b7622a271-istiod-ca-cert\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.230128 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.230098    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-workload-socket\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.230204 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.230181    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-credential-socket\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.230438 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.230413    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-workload-certs\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.232572 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.232504    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-istio-envoy\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.232710 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.232685    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04b5051c-1bff-4550-b442-7f0b7622a271-istio-podinfo\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.306506 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.292322    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04b5051c-1bff-4550-b442-7f0b7622a271-istio-token\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.306506 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.292868    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk6j6\" (UniqueName: \"kubernetes.io/projected/04b5051c-1bff-4550-b442-7f0b7622a271-kube-api-access-xk6j6\") pod \"e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") " pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.350199 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.350168    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:13.809705 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.809600    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"]
Oct 30 20:21:13.812674 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:21:13.812644    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b5051c_1bff_4550_b442_7f0b7622a271.slice/crio-11b1cb1a72d75178197d1dd5c083d786688c0dc4d58aa556e3a3b296fe66bc53 WatchSource:0}: Error finding container 11b1cb1a72d75178197d1dd5c083d786688c0dc4d58aa556e3a3b296fe66bc53: Status 404 returned error can't find the container with id 11b1cb1a72d75178197d1dd5c083d786688c0dc4d58aa556e3a3b296fe66bc53
Oct 30 20:21:13.815234 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.815199    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:21:13.815303 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.815267    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:21:13.815303 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:13.815299    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:21:14.133881 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:14.133842    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7" event={"ID":"04b5051c-1bff-4550-b442-7f0b7622a271","Type":"ContainerStarted","Data":"265b226f7c326898944e7502af9464b56f87ce3e1929059ef9f48424883a29b9"}
Oct 30 20:21:14.134291 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:14.133906    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7" event={"ID":"04b5051c-1bff-4550-b442-7f0b7622a271","Type":"ContainerStarted","Data":"11b1cb1a72d75178197d1dd5c083d786688c0dc4d58aa556e3a3b296fe66bc53"}
Oct 30 20:21:14.258315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:14.258261    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7" podStartSLOduration=2.25824741 podStartE2EDuration="2.25824741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:14.254793441 +0000 UTC m=+2312.987008833" watchObservedRunningTime="2026-10-30 20:21:14.25824741 +0000 UTC m=+2312.990462799"
Oct 30 20:21:14.350480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:14.350420    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:14.356290 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:14.356264    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:15.140608 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:15.140576    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:15.141503 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:15.141480    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:21:17.842159 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:17.842123    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt"]
Oct 30 20:21:17.842569 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:17.842331    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt" podUID="e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb" containerName="maas-api" containerID="cri-o://3c920dded65b63289c86c65cdabdf68a864f32f930c720908c00d652c82077cc" gracePeriod=30
Oct 30 20:21:18.124054 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:18.124024    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt"
Oct 30 20:21:18.160089 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:18.160043    5958 generic.go:356] "Generic (PLEG): container finished" podID="e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb" containerID="3c920dded65b63289c86c65cdabdf68a864f32f930c720908c00d652c82077cc" exitCode=0
Oct 30 20:21:18.160248 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:18.160099    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt"
Oct 30 20:21:18.160248 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:18.160112    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt" event={"ID":"e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb","Type":"ContainerDied","Data":"3c920dded65b63289c86c65cdabdf68a864f32f930c720908c00d652c82077cc"}
Oct 30 20:21:18.160248 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:18.160176    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt" event={"ID":"e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb","Type":"ContainerDied","Data":"f78875b08d9932c9be49488cc17023d7a34321440d97754195c9b317071d9e20"}
Oct 30 20:21:18.160248 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:18.160199    5958 scope.go:122] "RemoveContainer" containerID="3c920dded65b63289c86c65cdabdf68a864f32f930c720908c00d652c82077cc"
Oct 30 20:21:18.170389 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:18.170360    5958 scope.go:122] "RemoveContainer" containerID="3c920dded65b63289c86c65cdabdf68a864f32f930c720908c00d652c82077cc"
Oct 30 20:21:18.170707 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:21:18.170681    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c920dded65b63289c86c65cdabdf68a864f32f930c720908c00d652c82077cc\": container with ID starting with 3c920dded65b63289c86c65cdabdf68a864f32f930c720908c00d652c82077cc not found: ID does not exist" containerID="3c920dded65b63289c86c65cdabdf68a864f32f930c720908c00d652c82077cc"
Oct 30 20:21:18.170774 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:18.170714    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c920dded65b63289c86c65cdabdf68a864f32f930c720908c00d652c82077cc"} err="failed to get container status \"3c920dded65b63289c86c65cdabdf68a864f32f930c720908c00d652c82077cc\": rpc error: code = NotFound desc = could not find container \"3c920dded65b63289c86c65cdabdf68a864f32f930c720908c00d652c82077cc\": container with ID starting with 3c920dded65b63289c86c65cdabdf68a864f32f930c720908c00d652c82077cc not found: ID does not exist"
Oct 30 20:21:18.197426 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:18.197392    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-metrics-certs\") pod \"e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb\" (UID: \"e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb\") "
Oct 30 20:21:18.197591 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:18.197442    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-kube-api-access-pzrhm\" (UniqueName: \"kubernetes.io/projected/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-kube-api-access-pzrhm\") pod \"e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb\" (UID: \"e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb\") "
Oct 30 20:21:18.197638 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:18.197596    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-openshift-service-ca\") pod \"e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb\" (UID: \"e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb\") "
Oct 30 20:21:18.197638 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:18.197619    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-maas-api-tls\") pod \"e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb\" (UID: \"e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb\") "
Oct 30 20:21:18.197966 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:18.197940    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-openshift-service-ca" pod "e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb" (UID: "e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:18.198258 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:18.198162    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-openshift-service-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:18.199718 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:18.199693    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-metrics-certs" pod "e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb" (UID: "e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:18.199818 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:18.199775    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-kube-api-access-pzrhm" pod "e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb" (UID: "e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb"). InnerVolumeSpecName "kube-api-access-pzrhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:18.199818 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:18.199779    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-maas-api-tls" pod "e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb" (UID: "e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:18.299240 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:18.299210    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-maas-api-tls\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:18.299240 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:18.299239    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-metrics-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:18.299416 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:18.299249    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzrhm\" (UniqueName: \"kubernetes.io/projected/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb-kube-api-access-pzrhm\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:18.584118 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:18.584084    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt"]
Oct 30 20:21:18.588041 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:18.588008    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a2103a-8699b9c8b4-j4mwt"]
Oct 30 20:21:19.405110 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:19.405072    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb" path="/var/lib/kubelet/pods/e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb/volumes"
Oct 30 20:21:20.984461 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:20.984419    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8"]
Oct 30 20:21:20.985436 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:20.985409    5958 cpu_manager.go:414] "containerMap: removing container" podUID="e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb" containerName="maas-api"
Oct 30 20:21:20.985601 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:20.985439    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb" containerName="maas-api"
Oct 30 20:21:20.985728 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:20.985713    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0f6b6ff-38d0-4f73-8b80-b9767a2a9ecb" containerName="maas-api"
Oct 30 20:21:20.991789 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:20.991764    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8"
Oct 30 20:21:21.041504 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:21.040985    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57472cf-a3f7-486d-9368-1c692e31bedc-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8\" (UID: \"f57472cf-a3f7-486d-9368-1c692e31bedc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8"
Oct 30 20:21:21.041504 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:21.041063    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfggd\" (UniqueName: \"kubernetes.io/projected/f57472cf-a3f7-486d-9368-1c692e31bedc-kube-api-access-wfggd\") pod \"payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8\" (UID: \"f57472cf-a3f7-486d-9368-1c692e31bedc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8"
Oct 30 20:21:21.046686 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:21.046638    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8"]
Oct 30 20:21:21.091181 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:21.091145    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a2386c-dockercfg-kf6lq\""
Oct 30 20:21:21.091181 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:21.091164    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a2386c\""
Oct 30 20:21:21.142435 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:21.142396    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57472cf-a3f7-486d-9368-1c692e31bedc-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8\" (UID: \"f57472cf-a3f7-486d-9368-1c692e31bedc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8"
Oct 30 20:21:21.142686 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:21.142486    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfggd\" (UniqueName: \"kubernetes.io/projected/f57472cf-a3f7-486d-9368-1c692e31bedc-kube-api-access-wfggd\") pod \"payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8\" (UID: \"f57472cf-a3f7-486d-9368-1c692e31bedc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8"
Oct 30 20:21:21.144071 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:21.144038    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57472cf-a3f7-486d-9368-1c692e31bedc-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8\" (UID: \"f57472cf-a3f7-486d-9368-1c692e31bedc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8"
Oct 30 20:21:21.204370 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:21.204340    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfggd\" (UniqueName: \"kubernetes.io/projected/f57472cf-a3f7-486d-9368-1c692e31bedc-kube-api-access-wfggd\") pod \"payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8\" (UID: \"f57472cf-a3f7-486d-9368-1c692e31bedc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8"
Oct 30 20:21:21.310715 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:21.310642    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8"
Oct 30 20:21:21.689440 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:21.689412    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8"]
Oct 30 20:21:21.691183 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:21:21.691157    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf57472cf_a3f7_486d_9368_1c692e31bedc.slice/crio-19776372fe411cfde39e9300933bbe917e57cdd7a965a3f8b000b20ea2e765d1 WatchSource:0}: Error finding container 19776372fe411cfde39e9300933bbe917e57cdd7a965a3f8b000b20ea2e765d1: Status 404 returned error can't find the container with id 19776372fe411cfde39e9300933bbe917e57cdd7a965a3f8b000b20ea2e765d1
Oct 30 20:21:22.189516 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:22.189478    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8" event={"ID":"f57472cf-a3f7-486d-9368-1c692e31bedc","Type":"ContainerStarted","Data":"c1ca32eecf12ef718a01082c8a61058a1d075de1dcec3fab494a87790fba2f30"}
Oct 30 20:21:22.189957 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:22.189540    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8" event={"ID":"f57472cf-a3f7-486d-9368-1c692e31bedc","Type":"ContainerStarted","Data":"19776372fe411cfde39e9300933bbe917e57cdd7a965a3f8b000b20ea2e765d1"}
Oct 30 20:21:22.189957 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:22.189628    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8"
Oct 30 20:21:22.313894 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:22.313840    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8" podStartSLOduration=2.313826122 podStartE2EDuration="2.313826122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:22.311265844 +0000 UTC m=+2321.043481235" watchObservedRunningTime="2026-10-30 20:21:22.313826122 +0000 UTC m=+2321.046041523"
Oct 30 20:21:33.198797 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:33.198768    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8"
Oct 30 20:21:34.280833 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:34.280787    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"]
Oct 30 20:21:34.981761 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:34.981722    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"]
Oct 30 20:21:34.985942 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:34.985915    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.031232 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.031206    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"]
Oct 30 20:21:35.035165 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.035145    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.080907 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.080876    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"]
Oct 30 20:21:35.105884 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.105857    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"]
Oct 30 20:21:35.139259 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.139223    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.139405 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.139309    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.139405 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.139351    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.139405 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.139400    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.139535 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.139509    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6jrh\" (UniqueName: \"kubernetes.io/projected/a993fc51-49ae-4c3d-8c26-1f256e5064ed-kube-api-access-v6jrh\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.139572 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.139558    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.139626 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.139610    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.139763 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.139736    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmr6l\" (UniqueName: \"kubernetes.io/projected/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-kube-api-access-lmr6l\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.139826 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.139810    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.139871 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.139852    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.139915 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.139899    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.139950 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.139941    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.139985 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.139959    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.140017 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.139996    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.140050 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.140026    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.140176 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.140161    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.140238 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.140224    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.140274 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.140256    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.241999 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.241929    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.241999 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.241984    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.242209 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.242030    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.242209 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.242068    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.242209 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.242111    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.242209 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.242177    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.242500 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.242213    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6jrh\" (UniqueName: \"kubernetes.io/projected/a993fc51-49ae-4c3d-8c26-1f256e5064ed-kube-api-access-v6jrh\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.242500 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.242243    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.242500 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.242277    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.242500 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.242337    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmr6l\" (UniqueName: \"kubernetes.io/projected/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-kube-api-access-lmr6l\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.242500 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.242393    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.242500 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.242414    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.242500 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.242422    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.242500 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.242446    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.242938 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.242522    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.242938 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.242544    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.242938 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.242568    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.242938 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.242600    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.242938 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.242675    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.242938 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.242778    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.242938 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.242782    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.242938 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.242915    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.243345 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.243002    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.243405 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.243380    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.243797 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.243527    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.243797 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.243731    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.243797 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.243781    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.243965 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.243855    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.244983 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.244956    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.245081 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.245046    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.245081 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.245048    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.245397 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.245382    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.305248 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.305220    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmr6l\" (UniqueName: \"kubernetes.io/projected/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-kube-api-access-lmr6l\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.305708 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.305290    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-nspll\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.305924 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.305896    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.306042 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.306027    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6jrh\" (UniqueName: \"kubernetes.io/projected/a993fc51-49ae-4c3d-8c26-1f256e5064ed-kube-api-access-v6jrh\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.348445 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.348412    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:35.600798 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.600764    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:35.781444 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.781418    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"]
Oct 30 20:21:35.783159 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:21:35.783128    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36638d21_d7dd_40ec_9a33_83dcbbe2d7ae.slice/crio-aada4b642a52c0e9226f0f761139459a00c78d179d81a064a7f1de420643d854 WatchSource:0}: Error finding container aada4b642a52c0e9226f0f761139459a00c78d179d81a064a7f1de420643d854: Status 404 returned error can't find the container with id aada4b642a52c0e9226f0f761139459a00c78d179d81a064a7f1de420643d854
Oct 30 20:21:35.786109 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.786075    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:21:35.786215 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.786148    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:21:35.786215 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.786193    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:21:35.982751 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.982722    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"]
Oct 30 20:21:35.983559 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:21:35.983526    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda993fc51_49ae_4c3d_8c26_1f256e5064ed.slice/crio-0bfed1831eaa8b3c4969774eb9c9f22f2e63791823dc31b5a3baf0a0976c77f9 WatchSource:0}: Error finding container 0bfed1831eaa8b3c4969774eb9c9f22f2e63791823dc31b5a3baf0a0976c77f9: Status 404 returned error can't find the container with id 0bfed1831eaa8b3c4969774eb9c9f22f2e63791823dc31b5a3baf0a0976c77f9
Oct 30 20:21:35.986061 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.986029    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:21:35.986129 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.986095    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:21:35.986129 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:35.986123    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:21:36.311664 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:36.311569    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m" event={"ID":"a993fc51-49ae-4c3d-8c26-1f256e5064ed","Type":"ContainerStarted","Data":"2cdeb0eda1225c96d3d10d62205122c338b46fefce991375cd1310195fba8403"}
Oct 30 20:21:36.311664 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:36.311635    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m" event={"ID":"a993fc51-49ae-4c3d-8c26-1f256e5064ed","Type":"ContainerStarted","Data":"0bfed1831eaa8b3c4969774eb9c9f22f2e63791823dc31b5a3baf0a0976c77f9"}
Oct 30 20:21:36.313481 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:36.313433    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll" event={"ID":"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae","Type":"ContainerStarted","Data":"d67212398c218a75ab46f08d283ed2b36dc8ca00a5d93f34cbd805cc14d10c16"}
Oct 30 20:21:36.313602 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:36.313497    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll" event={"ID":"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae","Type":"ContainerStarted","Data":"aada4b642a52c0e9226f0f761139459a00c78d179d81a064a7f1de420643d854"}
Oct 30 20:21:36.349275 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:36.349248    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:36.350963 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:36.350936    5958 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 30 20:21:36.351075 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:36.350981    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll" podUID="36638d21-d7dd-40ec-9a33-83dcbbe2d7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 30 20:21:36.433636 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:36.433563    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m" podStartSLOduration=2.433545494 podStartE2EDuration="2.433545494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:36.432586625 +0000 UTC m=+2335.164802015" watchObservedRunningTime="2026-10-30 20:21:36.433545494 +0000 UTC m=+2335.165760886"
Oct 30 20:21:36.556753 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:36.556704    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll" podStartSLOduration=2.55668837 podStartE2EDuration="2.55668837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:36.553956761 +0000 UTC m=+2335.286172150" watchObservedRunningTime="2026-10-30 20:21:36.55668837 +0000 UTC m=+2335.288903759"
Oct 30 20:21:36.601410 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:36.601382    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:36.606849 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:21:36.606823    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:21:37.362478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:37.362054    5958 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 30 20:21:37.362478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:37.362154    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll" podUID="36638d21-d7dd-40ec-9a33-83dcbbe2d7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 30 20:21:37.603016 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:37.602977    5958 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:21:37.603206 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:37.603034    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m" podUID="a993fc51-49ae-4c3d-8c26-1f256e5064ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 20:21:38.357477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:38.356855    5958 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 30 20:21:38.357477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:38.356918    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll" podUID="36638d21-d7dd-40ec-9a33-83dcbbe2d7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 30 20:21:38.601285 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:38.601247    5958 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:21:38.601718 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:38.601305    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m" podUID="a993fc51-49ae-4c3d-8c26-1f256e5064ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 20:21:39.358474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:39.358018    5958 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 30 20:21:39.358474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:39.358082    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll" podUID="36638d21-d7dd-40ec-9a33-83dcbbe2d7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 30 20:21:39.601422 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:39.601376    5958 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:21:39.601951 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:39.601433    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m" podUID="a993fc51-49ae-4c3d-8c26-1f256e5064ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 20:21:40.097551 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:40.097502    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"]
Oct 30 20:21:40.097745 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:40.097724    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6" podUID="f382f3fe-b7ff-4e1e-a36b-98b8b0882937" containerName="istio-proxy" containerID="cri-o://d89cb5e7a852955573bbc24bfdc30e7f2583e24f905dde6a06e793a59659fe23" gracePeriod=30
Oct 30 20:21:40.379533 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:40.358017    5958 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 30 20:21:40.379533 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:40.358527    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll" podUID="36638d21-d7dd-40ec-9a33-83dcbbe2d7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 30 20:21:40.601880 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:40.601843    5958 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:21:40.602263 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:40.601900    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m" podUID="a993fc51-49ae-4c3d-8c26-1f256e5064ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 20:21:40.896500 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:40.896465    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7"]
Oct 30 20:21:41.026633 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:41.026577    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9"]
Oct 30 20:21:41.349074 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:41.349028    5958 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 30 20:21:41.349403 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:41.349351    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll" podUID="36638d21-d7dd-40ec-9a33-83dcbbe2d7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 30 20:21:41.604070 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:41.603983    5958 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:21:41.604698 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:41.604663    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m" podUID="a993fc51-49ae-4c3d-8c26-1f256e5064ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 20:21:42.358474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:42.357869    5958 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Oct 30 20:21:42.358474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:42.357953    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll" podUID="36638d21-d7dd-40ec-9a33-83dcbbe2d7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Oct 30 20:21:42.602204 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:42.602117    5958 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 20:21:42.602580 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:42.602548    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m" podUID="a993fc51-49ae-4c3d-8c26-1f256e5064ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 20:21:43.353622 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:43.353592    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:43.354033 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:43.354021    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:43.354823 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:43.354801    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:21:43.606180 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:43.606109    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:43.606438 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:43.606408    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:43.607124 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:43.607106    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:21:45.377937 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.377879    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:21:45.404781 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.404724    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-kube-api-access-qpwbt\" (UniqueName: \"kubernetes.io/projected/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-kube-api-access-qpwbt\") pod \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") "
Oct 30 20:21:45.404781 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.404763    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-workload-socket\") pod \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") "
Oct 30 20:21:45.404964 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.404843    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istiod-ca-cert\") pod \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") "
Oct 30 20:21:45.404964 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.404873    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-data\") pod \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") "
Oct 30 20:21:45.404964 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.404914    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-credential-socket\") pod \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") "
Oct 30 20:21:45.405081 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.404984    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-token\" (UniqueName: \"kubernetes.io/projected/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-token\") pod \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") "
Oct 30 20:21:45.405081 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.405028    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-workload-certs\") pod \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") "
Oct 30 20:21:45.405222 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.405199    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-envoy\") pod \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") "
Oct 30 20:21:45.405596 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.405320    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-workload-socket" pod "f382f3fe-b7ff-4e1e-a36b-98b8b0882937" (UID: "f382f3fe-b7ff-4e1e-a36b-98b8b0882937"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:45.405596 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.405329    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-podinfo\") pod \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\" (UID: \"f382f3fe-b7ff-4e1e-a36b-98b8b0882937\") "
Oct 30 20:21:45.405794 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.405690    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-data" pod "f382f3fe-b7ff-4e1e-a36b-98b8b0882937" (UID: "f382f3fe-b7ff-4e1e-a36b-98b8b0882937"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:45.405889 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.405870    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-workload-certs" pod "f382f3fe-b7ff-4e1e-a36b-98b8b0882937" (UID: "f382f3fe-b7ff-4e1e-a36b-98b8b0882937"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:45.406024 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.406003    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.406152 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.406028    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.406152 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.406043    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.406152 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.406119    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-credential-socket" pod "f382f3fe-b7ff-4e1e-a36b-98b8b0882937" (UID: "f382f3fe-b7ff-4e1e-a36b-98b8b0882937"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:45.406354 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.406337    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istiod-ca-cert" pod "f382f3fe-b7ff-4e1e-a36b-98b8b0882937" (UID: "f382f3fe-b7ff-4e1e-a36b-98b8b0882937"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:45.407618 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.407580    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-kube-api-access-qpwbt" pod "f382f3fe-b7ff-4e1e-a36b-98b8b0882937" (UID: "f382f3fe-b7ff-4e1e-a36b-98b8b0882937"). InnerVolumeSpecName "kube-api-access-qpwbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:45.408666 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.408640    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-podinfo" pod "f382f3fe-b7ff-4e1e-a36b-98b8b0882937" (UID: "f382f3fe-b7ff-4e1e-a36b-98b8b0882937"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:21:45.408836 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.408813    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-token" pod "f382f3fe-b7ff-4e1e-a36b-98b8b0882937" (UID: "f382f3fe-b7ff-4e1e-a36b-98b8b0882937"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:45.409503 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.409480    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-envoy" pod "f382f3fe-b7ff-4e1e-a36b-98b8b0882937" (UID: "f382f3fe-b7ff-4e1e-a36b-98b8b0882937"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:45.507921 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.507887    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.507921 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.507920    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.508071 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.507934    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.508071 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.507948    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpwbt\" (UniqueName: \"kubernetes.io/projected/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-kube-api-access-qpwbt\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.508071 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.507962    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.508071 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.507975    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f382f3fe-b7ff-4e1e-a36b-98b8b0882937-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:45.547949 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.547915    5958 generic.go:356] "Generic (PLEG): container finished" podID="f382f3fe-b7ff-4e1e-a36b-98b8b0882937" containerID="d89cb5e7a852955573bbc24bfdc30e7f2583e24f905dde6a06e793a59659fe23" exitCode=0
Oct 30 20:21:45.548106 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.548063    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"
Oct 30 20:21:45.548106 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.548066    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6" event={"ID":"f382f3fe-b7ff-4e1e-a36b-98b8b0882937","Type":"ContainerDied","Data":"d89cb5e7a852955573bbc24bfdc30e7f2583e24f905dde6a06e793a59659fe23"}
Oct 30 20:21:45.548234 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.548129    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6" event={"ID":"f382f3fe-b7ff-4e1e-a36b-98b8b0882937","Type":"ContainerDied","Data":"cf8c33d21babc7be4b182c47952f806dbcfcb0ef937f6efbdb8040e22b6f3485"}
Oct 30 20:21:45.548234 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.548163    5958 scope.go:122] "RemoveContainer" containerID="d89cb5e7a852955573bbc24bfdc30e7f2583e24f905dde6a06e793a59659fe23"
Oct 30 20:21:45.559233 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.559215    5958 scope.go:122] "RemoveContainer" containerID="d89cb5e7a852955573bbc24bfdc30e7f2583e24f905dde6a06e793a59659fe23"
Oct 30 20:21:45.559509 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:21:45.559489    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d89cb5e7a852955573bbc24bfdc30e7f2583e24f905dde6a06e793a59659fe23\": container with ID starting with d89cb5e7a852955573bbc24bfdc30e7f2583e24f905dde6a06e793a59659fe23 not found: ID does not exist" containerID="d89cb5e7a852955573bbc24bfdc30e7f2583e24f905dde6a06e793a59659fe23"
Oct 30 20:21:45.559575 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.559521    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89cb5e7a852955573bbc24bfdc30e7f2583e24f905dde6a06e793a59659fe23"} err="failed to get container status \"d89cb5e7a852955573bbc24bfdc30e7f2583e24f905dde6a06e793a59659fe23\": rpc error: code = NotFound desc = could not find container \"d89cb5e7a852955573bbc24bfdc30e7f2583e24f905dde6a06e793a59659fe23\": container with ID starting with d89cb5e7a852955573bbc24bfdc30e7f2583e24f905dde6a06e793a59659fe23 not found: ID does not exist"
Oct 30 20:21:45.714533 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.714464    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"]
Oct 30 20:21:45.718803 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.718773    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-j2pk6"]
Oct 30 20:21:45.897536 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:45.897497    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7" podUID="ecb7666c-9fbc-4d05-942b-de443eef8f24" containerName="payload-processing" containerID="cri-o://236e59c673a70ccb8fb878c8b08cce59158415b6c08f9b9376f83f7c6552296e" gracePeriod=25
Oct 30 20:21:46.027142 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:46.027062    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9" podUID="8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7" containerName="payload-pre-processing" containerID="cri-o://78c651d3fe63278b6163ca4afbb6cb89eaa1484d9188f7f3988edcbf28237f3b" gracePeriod=25
Oct 30 20:21:47.407017 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:47.406980    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f382f3fe-b7ff-4e1e-a36b-98b8b0882937" path="/var/lib/kubelet/pods/f382f3fe-b7ff-4e1e-a36b-98b8b0882937/volumes"
Oct 30 20:21:49.316266 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:49.316208    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main" containerID="cri-o://80855c98ca95a234482722c9e14a075988a01af2bab1918f541d3a01b0b8e254" gracePeriod=45
Oct 30 20:21:49.582795 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:49.582721    5958 generic.go:356] "Generic (PLEG): container finished" podID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerID="80855c98ca95a234482722c9e14a075988a01af2bab1918f541d3a01b0b8e254" exitCode=0
Oct 30 20:21:49.582922 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:49.582795    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" event={"ID":"bf720d46-c112-42e5-8a6b-11c6cb96042d","Type":"ContainerDied","Data":"80855c98ca95a234482722c9e14a075988a01af2bab1918f541d3a01b0b8e254"}
Oct 30 20:21:49.582922 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:49.582843    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb" event={"ID":"bf720d46-c112-42e5-8a6b-11c6cb96042d","Type":"ContainerDied","Data":"a3501669b3a02645d50946f0531b5feffefaeacab00ee83b7774ad1d03d1b105"}
Oct 30 20:21:49.582922 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:49.582855    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3501669b3a02645d50946f0531b5feffefaeacab00ee83b7774ad1d03d1b105"
Oct 30 20:21:49.584873 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:49.584858    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:21:49.666740 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:49.666708    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-home\" (UniqueName: \"kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-home\") pod \"bf720d46-c112-42e5-8a6b-11c6cb96042d\" (UID: \"bf720d46-c112-42e5-8a6b-11c6cb96042d\") "
Oct 30 20:21:49.666740 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:49.666752    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf720d46-c112-42e5-8a6b-11c6cb96042d-kube-api-access-wtwqw\" (UniqueName: \"kubernetes.io/projected/bf720d46-c112-42e5-8a6b-11c6cb96042d-kube-api-access-wtwqw\") pod \"bf720d46-c112-42e5-8a6b-11c6cb96042d\" (UID: \"bf720d46-c112-42e5-8a6b-11c6cb96042d\") "
Oct 30 20:21:49.666980 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:49.666817    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf720d46-c112-42e5-8a6b-11c6cb96042d-tls-certs\" (UniqueName: \"kubernetes.io/secret/bf720d46-c112-42e5-8a6b-11c6cb96042d-tls-certs\") pod \"bf720d46-c112-42e5-8a6b-11c6cb96042d\" (UID: \"bf720d46-c112-42e5-8a6b-11c6cb96042d\") "
Oct 30 20:21:49.666980 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:49.666949    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-model-cache\") pod \"bf720d46-c112-42e5-8a6b-11c6cb96042d\" (UID: \"bf720d46-c112-42e5-8a6b-11c6cb96042d\") "
Oct 30 20:21:49.666980 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:49.666972    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-home" pod "bf720d46-c112-42e5-8a6b-11c6cb96042d" (UID: "bf720d46-c112-42e5-8a6b-11c6cb96042d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:49.667091 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:49.667029    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-tmp-dir\") pod \"bf720d46-c112-42e5-8a6b-11c6cb96042d\" (UID: \"bf720d46-c112-42e5-8a6b-11c6cb96042d\") "
Oct 30 20:21:49.667091 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:49.667074    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-dshm\") pod \"bf720d46-c112-42e5-8a6b-11c6cb96042d\" (UID: \"bf720d46-c112-42e5-8a6b-11c6cb96042d\") "
Oct 30 20:21:49.667239 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:49.667205    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-model-cache" pod "bf720d46-c112-42e5-8a6b-11c6cb96042d" (UID: "bf720d46-c112-42e5-8a6b-11c6cb96042d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:49.667326 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:49.667295    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-tmp-dir" pod "bf720d46-c112-42e5-8a6b-11c6cb96042d" (UID: "bf720d46-c112-42e5-8a6b-11c6cb96042d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:49.667671 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:49.667653    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-tmp-dir\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:49.667761 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:49.667676    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-home\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:49.667761 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:49.667690    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-model-cache\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:49.669220 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:49.669200    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf720d46-c112-42e5-8a6b-11c6cb96042d-tls-certs" pod "bf720d46-c112-42e5-8a6b-11c6cb96042d" (UID: "bf720d46-c112-42e5-8a6b-11c6cb96042d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:49.669321 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:49.669246    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf720d46-c112-42e5-8a6b-11c6cb96042d-kube-api-access-wtwqw" pod "bf720d46-c112-42e5-8a6b-11c6cb96042d" (UID: "bf720d46-c112-42e5-8a6b-11c6cb96042d"). InnerVolumeSpecName "kube-api-access-wtwqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:49.669321 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:49.669246    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-dshm" pod "bf720d46-c112-42e5-8a6b-11c6cb96042d" (UID: "bf720d46-c112-42e5-8a6b-11c6cb96042d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:21:49.768924 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:49.768843    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf720d46-c112-42e5-8a6b-11c6cb96042d-tls-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:49.768924 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:49.768877    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf720d46-c112-42e5-8a6b-11c6cb96042d-dshm\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:49.768924 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:49.768887    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtwqw\" (UniqueName: \"kubernetes.io/projected/bf720d46-c112-42e5-8a6b-11c6cb96042d-kube-api-access-wtwqw\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:50.146175 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.146142    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"]
Oct 30 20:21:50.146793 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.146777    5958 cpu_manager.go:414] "containerMap: removing container" podUID="f382f3fe-b7ff-4e1e-a36b-98b8b0882937" containerName="istio-proxy"
Oct 30 20:21:50.146843 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.146794    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f382f3fe-b7ff-4e1e-a36b-98b8b0882937" containerName="istio-proxy"
Oct 30 20:21:50.146843 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.146829    5958 cpu_manager.go:414] "containerMap: removing container" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main"
Oct 30 20:21:50.146843 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.146834    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main"
Oct 30 20:21:50.146939 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.146929    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" containerName="main"
Oct 30 20:21:50.146977 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.146947    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="f382f3fe-b7ff-4e1e-a36b-98b8b0882937" containerName="istio-proxy"
Oct 30 20:21:50.153895 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.153854    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.180130 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.180100    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.180261 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.180240    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.180306 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.180295    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.180427 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.180406    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.180497 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.180483    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p24b\" (UniqueName: \"kubernetes.io/projected/68f462dc-feb6-44af-a8af-bc45ce2e0e77-kube-api-access-9p24b\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.180548 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.180515    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.180584 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.180556    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.180629 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.180617    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.180663 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.180641    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.212709 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.212676    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"]
Oct 30 20:21:50.282064 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.282024    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.282247 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.282074    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p24b\" (UniqueName: \"kubernetes.io/projected/68f462dc-feb6-44af-a8af-bc45ce2e0e77-kube-api-access-9p24b\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.282247 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.282116    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.282247 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.282139    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.282247 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.282217    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.282247 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.282244    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.282564 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.282294    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.282564 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.282465    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.282669 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.282560    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.282717 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.282662    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-workload-certs\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.282717 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.282684    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-workload-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.282899 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.282873    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-credential-socket\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.282958 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.282899    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-data\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.283318 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.283296    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istiod-ca-cert\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.284475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.284443    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-envoy\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.284755 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.284738    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-podinfo\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.344071 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.344043    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-token\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.344411 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.344348    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p24b\" (UniqueName: \"kubernetes.io/projected/68f462dc-feb6-44af-a8af-bc45ce2e0e77-kube-api-access-9p24b\") pod \"e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") " pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.472116 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.472028    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:50.590862 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.590834    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"
Oct 30 20:21:50.756376 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.756289    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"]
Oct 30 20:21:50.768214 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.768161    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-d67ea9/distinct-w2-d67ea9-kserve-6c94f795bc-5g4bb"]
Oct 30 20:21:50.924771 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.924741    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"]
Oct 30 20:21:50.925860 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:21:50.925821    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f462dc_feb6_44af_a8af_bc45ce2e0e77.slice/crio-6e4b0ca3a552cfecb05ee5494a09297658740ac46165aa5550d301d1920f3227 WatchSource:0}: Error finding container 6e4b0ca3a552cfecb05ee5494a09297658740ac46165aa5550d301d1920f3227: Status 404 returned error can't find the container with id 6e4b0ca3a552cfecb05ee5494a09297658740ac46165aa5550d301d1920f3227
Oct 30 20:21:50.928507 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.928473    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:21:50.928587 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.928541    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:21:50.928587 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:50.928569    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:21:51.401025 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:51.400989    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf720d46-c112-42e5-8a6b-11c6cb96042d" path="/var/lib/kubelet/pods/bf720d46-c112-42e5-8a6b-11c6cb96042d/volumes"
Oct 30 20:21:51.615534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:51.615487    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9" event={"ID":"68f462dc-feb6-44af-a8af-bc45ce2e0e77","Type":"ContainerStarted","Data":"4f904325479799b0c3a1d8f710a7c52f4d4aeab32fda4ddf3467a42a622eaf6f"}
Oct 30 20:21:51.615721 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:51.615562    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9" event={"ID":"68f462dc-feb6-44af-a8af-bc45ce2e0e77","Type":"ContainerStarted","Data":"6e4b0ca3a552cfecb05ee5494a09297658740ac46165aa5550d301d1920f3227"}
Oct 30 20:21:51.784598 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:51.784440    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9" podStartSLOduration=1.7844238190000001 podStartE2EDuration="1.784423819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:51.781812486 +0000 UTC m=+2350.514027877" watchObservedRunningTime="2026-10-30 20:21:51.784423819 +0000 UTC m=+2350.516639208"
Oct 30 20:21:52.472499 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:52.472434    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:52.474225 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:52.474197    5958 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 20:21:52.474321 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:52.474270    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9" podUID="68f462dc-feb6-44af-a8af-bc45ce2e0e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 20:21:53.472630 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:53.472589    5958 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 20:21:53.473003 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:53.472663    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9" podUID="68f462dc-feb6-44af-a8af-bc45ce2e0e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 20:21:54.409886 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:54.409836    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9" podUID="8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:21:54.409886 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:54.409872    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7" podUID="ecb7666c-9fbc-4d05-942b-de443eef8f24" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:21:54.472764 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:54.472725    5958 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 20:21:54.473138 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:54.472793    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9" podUID="68f462dc-feb6-44af-a8af-bc45ce2e0e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 20:21:55.473215 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:55.473152    5958 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 20:21:55.473613 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:55.473258    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9" podUID="68f462dc-feb6-44af-a8af-bc45ce2e0e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 20:21:56.472686 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:56.472652    5958 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 20:21:56.472980 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:56.472944    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9" podUID="68f462dc-feb6-44af-a8af-bc45ce2e0e77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 20:21:58.016692 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:58.016660    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:58.017872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:58.017843    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:21:58.019008 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:21:58.018885    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:22:04.410049 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:04.410010    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7" podUID="ecb7666c-9fbc-4d05-942b-de443eef8f24" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:22:04.410444 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:04.410010    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9" podUID="8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:22:06.174439 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.174383    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7"
Oct 30 20:22:06.283831 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.283805    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecb7666c-9fbc-4d05-942b-de443eef8f24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb7666c-9fbc-4d05-942b-de443eef8f24-plugins-config-volume\") pod \"ecb7666c-9fbc-4d05-942b-de443eef8f24\" (UID: \"ecb7666c-9fbc-4d05-942b-de443eef8f24\") "
Oct 30 20:22:06.283996 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.283972    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecb7666c-9fbc-4d05-942b-de443eef8f24-kube-api-access-qtdc7\" (UniqueName: \"kubernetes.io/projected/ecb7666c-9fbc-4d05-942b-de443eef8f24-kube-api-access-qtdc7\") pod \"ecb7666c-9fbc-4d05-942b-de443eef8f24\" (UID: \"ecb7666c-9fbc-4d05-942b-de443eef8f24\") "
Oct 30 20:22:06.284212 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.284185    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb7666c-9fbc-4d05-942b-de443eef8f24-plugins-config-volume" pod "ecb7666c-9fbc-4d05-942b-de443eef8f24" (UID: "ecb7666c-9fbc-4d05-942b-de443eef8f24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:06.284667 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.284578    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb7666c-9fbc-4d05-942b-de443eef8f24-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:06.286290 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.286267    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb7666c-9fbc-4d05-942b-de443eef8f24-kube-api-access-qtdc7" pod "ecb7666c-9fbc-4d05-942b-de443eef8f24" (UID: "ecb7666c-9fbc-4d05-942b-de443eef8f24"). InnerVolumeSpecName "kube-api-access-qtdc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:06.314436 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.314402    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9"
Oct 30 20:22:06.385956 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.385928    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7-kube-api-access-lqfkx\" (UniqueName: \"kubernetes.io/projected/8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7-kube-api-access-lqfkx\") pod \"8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7\" (UID: \"8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7\") "
Oct 30 20:22:06.386165 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.386145    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7-plugins-config-volume\") pod \"8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7\" (UID: \"8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7\") "
Oct 30 20:22:06.386579 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.386556    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7-plugins-config-volume" pod "8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7" (UID: "8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:06.386760 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.386745    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtdc7\" (UniqueName: \"kubernetes.io/projected/ecb7666c-9fbc-4d05-942b-de443eef8f24-kube-api-access-qtdc7\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:06.386806 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.386764    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:06.388209 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.388190    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7-kube-api-access-lqfkx" pod "8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7" (UID: "8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7"). InnerVolumeSpecName "kube-api-access-lqfkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:06.488619 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.488553    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqfkx\" (UniqueName: \"kubernetes.io/projected/8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7-kube-api-access-lqfkx\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:06.602765 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.602715    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"]
Oct 30 20:22:06.603268 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.603222    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6" podUID="6bc60e1a-391d-43b0-8b8e-2f6f33c234be" containerName="istio-proxy" containerID="cri-o://31ab70206f3a82a88027c08ead3041b13125165887f999631d71abfdc35bdfa6" gracePeriod=30
Oct 30 20:22:06.619417 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:22:06.619390    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:22:06.802051 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.801969    5958 generic.go:356] "Generic (PLEG): container finished" podID="8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7" containerID="78c651d3fe63278b6163ca4afbb6cb89eaa1484d9188f7f3988edcbf28237f3b" exitCode=0
Oct 30 20:22:06.802225 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.802048    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9" event={"ID":"8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7","Type":"ContainerDied","Data":"78c651d3fe63278b6163ca4afbb6cb89eaa1484d9188f7f3988edcbf28237f3b"}
Oct 30 20:22:06.802225 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.802102    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9" event={"ID":"8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7","Type":"ContainerDied","Data":"9349b99d75e42be2655e0bc9daabfaafb32b0700ca93f674ac66979663044519"}
Oct 30 20:22:06.802225 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.802132    5958 scope.go:122] "RemoveContainer" containerID="78c651d3fe63278b6163ca4afbb6cb89eaa1484d9188f7f3988edcbf28237f3b"
Oct 30 20:22:06.802521 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.802485    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9"
Oct 30 20:22:06.803799 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.803710    5958 generic.go:356] "Generic (PLEG): container finished" podID="ecb7666c-9fbc-4d05-942b-de443eef8f24" containerID="236e59c673a70ccb8fb878c8b08cce59158415b6c08f9b9376f83f7c6552296e" exitCode=0
Oct 30 20:22:06.803799 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.803790    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7" event={"ID":"ecb7666c-9fbc-4d05-942b-de443eef8f24","Type":"ContainerDied","Data":"236e59c673a70ccb8fb878c8b08cce59158415b6c08f9b9376f83f7c6552296e"}
Oct 30 20:22:06.803932 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.803829    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7" event={"ID":"ecb7666c-9fbc-4d05-942b-de443eef8f24","Type":"ContainerDied","Data":"28ef11dfa4b3b564b4d6b188cb82aebfdaee6feffdfb53949b03f80bd0efc306"}
Oct 30 20:22:06.803932 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.803796    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7"
Oct 30 20:22:06.815010 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.814989    5958 scope.go:122] "RemoveContainer" containerID="78c651d3fe63278b6163ca4afbb6cb89eaa1484d9188f7f3988edcbf28237f3b"
Oct 30 20:22:06.815372 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:22:06.815317    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78c651d3fe63278b6163ca4afbb6cb89eaa1484d9188f7f3988edcbf28237f3b\": container with ID starting with 78c651d3fe63278b6163ca4afbb6cb89eaa1484d9188f7f3988edcbf28237f3b not found: ID does not exist" containerID="78c651d3fe63278b6163ca4afbb6cb89eaa1484d9188f7f3988edcbf28237f3b"
Oct 30 20:22:06.815429 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.815384    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78c651d3fe63278b6163ca4afbb6cb89eaa1484d9188f7f3988edcbf28237f3b"} err="failed to get container status \"78c651d3fe63278b6163ca4afbb6cb89eaa1484d9188f7f3988edcbf28237f3b\": rpc error: code = NotFound desc = could not find container \"78c651d3fe63278b6163ca4afbb6cb89eaa1484d9188f7f3988edcbf28237f3b\": container with ID starting with 78c651d3fe63278b6163ca4afbb6cb89eaa1484d9188f7f3988edcbf28237f3b not found: ID does not exist"
Oct 30 20:22:06.815429 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.815409    5958 scope.go:122] "RemoveContainer" containerID="236e59c673a70ccb8fb878c8b08cce59158415b6c08f9b9376f83f7c6552296e"
Oct 30 20:22:06.824358 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.824343    5958 scope.go:122] "RemoveContainer" containerID="236e59c673a70ccb8fb878c8b08cce59158415b6c08f9b9376f83f7c6552296e"
Oct 30 20:22:06.824621 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:22:06.824600    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"236e59c673a70ccb8fb878c8b08cce59158415b6c08f9b9376f83f7c6552296e\": container with ID starting with 236e59c673a70ccb8fb878c8b08cce59158415b6c08f9b9376f83f7c6552296e not found: ID does not exist" containerID="236e59c673a70ccb8fb878c8b08cce59158415b6c08f9b9376f83f7c6552296e"
Oct 30 20:22:06.824663 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.824628    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"236e59c673a70ccb8fb878c8b08cce59158415b6c08f9b9376f83f7c6552296e"} err="failed to get container status \"236e59c673a70ccb8fb878c8b08cce59158415b6c08f9b9376f83f7c6552296e\": rpc error: code = NotFound desc = could not find container \"236e59c673a70ccb8fb878c8b08cce59158415b6c08f9b9376f83f7c6552296e\": container with ID starting with 236e59c673a70ccb8fb878c8b08cce59158415b6c08f9b9376f83f7c6552296e not found: ID does not exist"
Oct 30 20:22:06.967235 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.967204    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9"]
Oct 30 20:22:06.971760 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:06.971731    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d67ea9-546568b975-zxnz9"]
Oct 30 20:22:07.081941 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:07.081914    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7"]
Oct 30 20:22:07.084671 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:07.084645    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d67ea9-69cbd4bdbb-gsnn7"]
Oct 30 20:22:07.399435 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:07.399351    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7" path="/var/lib/kubelet/pods/8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7/volumes"
Oct 30 20:22:07.399979 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:07.399958    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecb7666c-9fbc-4d05-942b-de443eef8f24" path="/var/lib/kubelet/pods/ecb7666c-9fbc-4d05-942b-de443eef8f24/volumes"
Oct 30 20:22:07.536915 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:07.536889    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-575b12ca-mlv88"]
Oct 30 20:22:07.537520 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:07.537504    5958 cpu_manager.go:414] "containerMap: removing container" podUID="8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7" containerName="payload-pre-processing"
Oct 30 20:22:07.537571 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:07.537523    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7" containerName="payload-pre-processing"
Oct 30 20:22:07.537571 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:07.537537    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ecb7666c-9fbc-4d05-942b-de443eef8f24" containerName="payload-processing"
Oct 30 20:22:07.537571 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:07.537542    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecb7666c-9fbc-4d05-942b-de443eef8f24" containerName="payload-processing"
Oct 30 20:22:07.537674 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:07.537657    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecb7666c-9fbc-4d05-942b-de443eef8f24" containerName="payload-processing"
Oct 30 20:22:07.537706 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:07.537675    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="8572d9b7-b8ce-4f88-bfd6-7ec0e27e9fa7" containerName="payload-pre-processing"
Oct 30 20:22:07.542553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:07.542532    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575b12ca-mlv88"
Oct 30 20:22:07.605232 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:07.605198    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-575b12ca-mlv88"]
Oct 30 20:22:07.605917 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:07.605889    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30153d8d-0b5e-4d18-a442-c82144050a85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-575b12ca-mlv88\" (UID: \"30153d8d-0b5e-4d18-a442-c82144050a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575b12ca-mlv88"
Oct 30 20:22:07.606050 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:07.605951    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30153d8d-0b5e-4d18-a442-c82144050a85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-575b12ca-mlv88\" (UID: \"30153d8d-0b5e-4d18-a442-c82144050a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575b12ca-mlv88"
Oct 30 20:22:07.606166 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:07.606141    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfzjm\" (UniqueName: \"kubernetes.io/projected/30153d8d-0b5e-4d18-a442-c82144050a85-kube-api-access-zfzjm\") pod \"maas-api-revoke-sub-575b12ca-mlv88\" (UID: \"30153d8d-0b5e-4d18-a442-c82144050a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575b12ca-mlv88"
Oct 30 20:22:07.643573 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:07.643540    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:22:07.643723 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:07.643701    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:22:07.707963 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:07.707900    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30153d8d-0b5e-4d18-a442-c82144050a85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-575b12ca-mlv88\" (UID: \"30153d8d-0b5e-4d18-a442-c82144050a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575b12ca-mlv88"
Oct 30 20:22:07.708082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:07.707965    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfzjm\" (UniqueName: \"kubernetes.io/projected/30153d8d-0b5e-4d18-a442-c82144050a85-kube-api-access-zfzjm\") pod \"maas-api-revoke-sub-575b12ca-mlv88\" (UID: \"30153d8d-0b5e-4d18-a442-c82144050a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575b12ca-mlv88"
Oct 30 20:22:07.708129 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:07.708087    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30153d8d-0b5e-4d18-a442-c82144050a85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-575b12ca-mlv88\" (UID: \"30153d8d-0b5e-4d18-a442-c82144050a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575b12ca-mlv88"
Oct 30 20:22:07.708664 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:07.708643    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30153d8d-0b5e-4d18-a442-c82144050a85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-575b12ca-mlv88\" (UID: \"30153d8d-0b5e-4d18-a442-c82144050a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575b12ca-mlv88"
Oct 30 20:22:07.710367 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:07.710346    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30153d8d-0b5e-4d18-a442-c82144050a85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-575b12ca-mlv88\" (UID: \"30153d8d-0b5e-4d18-a442-c82144050a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575b12ca-mlv88"
Oct 30 20:22:07.771054 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:07.771029    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfzjm\" (UniqueName: \"kubernetes.io/projected/30153d8d-0b5e-4d18-a442-c82144050a85-kube-api-access-zfzjm\") pod \"maas-api-revoke-sub-575b12ca-mlv88\" (UID: \"30153d8d-0b5e-4d18-a442-c82144050a85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-575b12ca-mlv88"
Oct 30 20:22:07.858421 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:07.858398    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575b12ca-mlv88"
Oct 30 20:22:08.240609 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:08.240580    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-575b12ca-mlv88"]
Oct 30 20:22:08.241677 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:22:08.241654    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30153d8d_0b5e_4d18_a442_c82144050a85.slice/crio-f005b71c4c88ff79e8cc4ee6c19e8343932c84771677496f639021e9cbaa030a WatchSource:0}: Error finding container f005b71c4c88ff79e8cc4ee6c19e8343932c84771677496f639021e9cbaa030a: Status 404 returned error can't find the container with id f005b71c4c88ff79e8cc4ee6c19e8343932c84771677496f639021e9cbaa030a
Oct 30 20:22:08.820910 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:08.820878    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575b12ca-mlv88" event={"ID":"30153d8d-0b5e-4d18-a442-c82144050a85","Type":"ContainerStarted","Data":"9e374ead97cf5a8f0e4fedbb1289c4b2d9f7d650b84fa83c4bff48b5c2e2be23"}
Oct 30 20:22:08.821298 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:08.820928    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575b12ca-mlv88" event={"ID":"30153d8d-0b5e-4d18-a442-c82144050a85","Type":"ContainerStarted","Data":"f005b71c4c88ff79e8cc4ee6c19e8343932c84771677496f639021e9cbaa030a"}
Oct 30 20:22:08.937762 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:08.937655    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575b12ca-mlv88" podStartSLOduration=1.937641234 podStartE2EDuration="1.937641234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:08.936777718 +0000 UTC m=+2367.668993108" watchObservedRunningTime="2026-10-30 20:22:08.937641234 +0000 UTC m=+2367.669856623"
Oct 30 20:22:09.829072 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:09.829039    5958 generic.go:356] "Generic (PLEG): container finished" podID="30153d8d-0b5e-4d18-a442-c82144050a85" containerID="9e374ead97cf5a8f0e4fedbb1289c4b2d9f7d650b84fa83c4bff48b5c2e2be23" exitCode=0
Oct 30 20:22:09.829548 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:09.829123    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575b12ca-mlv88" event={"ID":"30153d8d-0b5e-4d18-a442-c82144050a85","Type":"ContainerDied","Data":"9e374ead97cf5a8f0e4fedbb1289c4b2d9f7d650b84fa83c4bff48b5c2e2be23"}
Oct 30 20:22:10.982558 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:10.982535    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575b12ca-mlv88"
Oct 30 20:22:11.159607 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.159426    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30153d8d-0b5e-4d18-a442-c82144050a85-kube-api-access-zfzjm\" (UniqueName: \"kubernetes.io/projected/30153d8d-0b5e-4d18-a442-c82144050a85-kube-api-access-zfzjm\") pod \"30153d8d-0b5e-4d18-a442-c82144050a85\" (UID: \"30153d8d-0b5e-4d18-a442-c82144050a85\") "
Oct 30 20:22:11.159607 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.159587    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30153d8d-0b5e-4d18-a442-c82144050a85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30153d8d-0b5e-4d18-a442-c82144050a85-maas-api-openshift-ca\") pod \"30153d8d-0b5e-4d18-a442-c82144050a85\" (UID: \"30153d8d-0b5e-4d18-a442-c82144050a85\") "
Oct 30 20:22:11.160130 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.160073    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30153d8d-0b5e-4d18-a442-c82144050a85-maas-api-openshift-ca" pod "30153d8d-0b5e-4d18-a442-c82144050a85" (UID: "30153d8d-0b5e-4d18-a442-c82144050a85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:11.161102 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.161054    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30153d8d-0b5e-4d18-a442-c82144050a85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30153d8d-0b5e-4d18-a442-c82144050a85-maas-api-xks-ca\") pod \"30153d8d-0b5e-4d18-a442-c82144050a85\" (UID: \"30153d8d-0b5e-4d18-a442-c82144050a85\") "
Oct 30 20:22:11.162034 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.162015    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30153d8d-0b5e-4d18-a442-c82144050a85-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:11.162675 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.162651    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30153d8d-0b5e-4d18-a442-c82144050a85-kube-api-access-zfzjm" pod "30153d8d-0b5e-4d18-a442-c82144050a85" (UID: "30153d8d-0b5e-4d18-a442-c82144050a85"). InnerVolumeSpecName "kube-api-access-zfzjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:11.163228 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.163207    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30153d8d-0b5e-4d18-a442-c82144050a85-maas-api-xks-ca" pod "30153d8d-0b5e-4d18-a442-c82144050a85" (UID: "30153d8d-0b5e-4d18-a442-c82144050a85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:22:11.263236 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.263198    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30153d8d-0b5e-4d18-a442-c82144050a85-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:11.263236 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.263228    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfzjm\" (UniqueName: \"kubernetes.io/projected/30153d8d-0b5e-4d18-a442-c82144050a85-kube-api-access-zfzjm\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:11.846576 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.846541    5958 generic.go:356] "Generic (PLEG): container finished" podID="6bc60e1a-391d-43b0-8b8e-2f6f33c234be" containerID="31ab70206f3a82a88027c08ead3041b13125165887f999631d71abfdc35bdfa6" exitCode=0
Oct 30 20:22:11.846755 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.846611    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6" event={"ID":"6bc60e1a-391d-43b0-8b8e-2f6f33c234be","Type":"ContainerDied","Data":"31ab70206f3a82a88027c08ead3041b13125165887f999631d71abfdc35bdfa6"}
Oct 30 20:22:11.846755 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.846708    5958 scope.go:122] "RemoveContainer" containerID="e05dc617cb1c861aedefcb6d4afe1c26b2a0a634e86006c15482f836f64e38a8"
Oct 30 20:22:11.849432 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.849358    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575b12ca-mlv88"
Oct 30 20:22:11.849684 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.849412    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-575b12ca-mlv88" event={"ID":"30153d8d-0b5e-4d18-a442-c82144050a85","Type":"ContainerDied","Data":"f005b71c4c88ff79e8cc4ee6c19e8343932c84771677496f639021e9cbaa030a"}
Oct 30 20:22:11.849684 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.849649    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f005b71c4c88ff79e8cc4ee6c19e8343932c84771677496f639021e9cbaa030a"
Oct 30 20:22:11.892043 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.892021    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 20:22:11.978296 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.978266    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-credential-socket\") pod \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") "
Oct 30 20:22:11.978414 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.978311    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-workload-socket\") pod \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") "
Oct 30 20:22:11.978414 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.978360    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-data\") pod \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") "
Oct 30 20:22:11.978414 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.978377    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-kube-api-access-jpq5x\" (UniqueName: \"kubernetes.io/projected/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-kube-api-access-jpq5x\") pod \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") "
Oct 30 20:22:11.978414 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.978396    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istiod-ca-cert\") pod \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") "
Oct 30 20:22:11.978686 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.978547    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-podinfo\") pod \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") "
Oct 30 20:22:11.978686 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.978591    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-credential-socket" pod "6bc60e1a-391d-43b0-8b8e-2f6f33c234be" (UID: "6bc60e1a-391d-43b0-8b8e-2f6f33c234be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:11.978686 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.978604    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-workload-certs\") pod \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") "
Oct 30 20:22:11.978829 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.978716    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-data" pod "6bc60e1a-391d-43b0-8b8e-2f6f33c234be" (UID: "6bc60e1a-391d-43b0-8b8e-2f6f33c234be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:11.978829 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.978736    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-workload-socket" pod "6bc60e1a-391d-43b0-8b8e-2f6f33c234be" (UID: "6bc60e1a-391d-43b0-8b8e-2f6f33c234be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:11.978829 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.978747    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-token\" (UniqueName: \"kubernetes.io/projected/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-token\") pod \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") "
Oct 30 20:22:11.978971 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.978849    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-envoy\") pod \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\" (UID: \"6bc60e1a-391d-43b0-8b8e-2f6f33c234be\") "
Oct 30 20:22:11.978971 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.978851    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-workload-certs" pod "6bc60e1a-391d-43b0-8b8e-2f6f33c234be" (UID: "6bc60e1a-391d-43b0-8b8e-2f6f33c234be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:11.978971 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.978876    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istiod-ca-cert" pod "6bc60e1a-391d-43b0-8b8e-2f6f33c234be" (UID: "6bc60e1a-391d-43b0-8b8e-2f6f33c234be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:11.979351 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.979326    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:11.979422 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.979353    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:11.979422 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.979367    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:11.979422 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.979379    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:11.979422 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.979392    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:11.980906 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.980879    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-envoy" pod "6bc60e1a-391d-43b0-8b8e-2f6f33c234be" (UID: "6bc60e1a-391d-43b0-8b8e-2f6f33c234be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:11.981021 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.980923    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-kube-api-access-jpq5x" pod "6bc60e1a-391d-43b0-8b8e-2f6f33c234be" (UID: "6bc60e1a-391d-43b0-8b8e-2f6f33c234be"). InnerVolumeSpecName "kube-api-access-jpq5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:11.981021 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.980933    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-token" pod "6bc60e1a-391d-43b0-8b8e-2f6f33c234be" (UID: "6bc60e1a-391d-43b0-8b8e-2f6f33c234be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:11.981021 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:11.980988    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-podinfo" pod "6bc60e1a-391d-43b0-8b8e-2f6f33c234be" (UID: "6bc60e1a-391d-43b0-8b8e-2f6f33c234be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:22:12.081510 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:12.081480    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpq5x\" (UniqueName: \"kubernetes.io/projected/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-kube-api-access-jpq5x\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:12.081867 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:12.081517    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:12.081867 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:12.081532    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:12.081867 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:12.081548    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc60e1a-391d-43b0-8b8e-2f6f33c234be-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:12.867556 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:12.862877    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6" event={"ID":"6bc60e1a-391d-43b0-8b8e-2f6f33c234be","Type":"ContainerDied","Data":"60ebb0273875ad49adbfa2cdec70b2063eedc4336a9df0d23562e7a5bff8b904"}
Oct 30 20:22:12.867556 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:12.862960    5958 scope.go:122] "RemoveContainer" containerID="31ab70206f3a82a88027c08ead3041b13125165887f999631d71abfdc35bdfa6"
Oct 30 20:22:12.867556 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:12.863233    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"
Oct 30 20:22:13.048944 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:13.048908    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"]
Oct 30 20:22:13.054385 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:13.054350    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f84d85-openshift-default-dfbcdd99c-k5nl6"]
Oct 30 20:22:13.399090 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:13.399057    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bc60e1a-391d-43b0-8b8e-2f6f33c234be" path="/var/lib/kubelet/pods/6bc60e1a-391d-43b0-8b8e-2f6f33c234be/volumes"
Oct 30 20:22:17.766052 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:17.766018    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986"]
Oct 30 20:22:17.766736 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:17.766718    5958 cpu_manager.go:414] "containerMap: removing container" podUID="6bc60e1a-391d-43b0-8b8e-2f6f33c234be" containerName="istio-proxy"
Oct 30 20:22:17.766736 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:17.766736    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc60e1a-391d-43b0-8b8e-2f6f33c234be" containerName="istio-proxy"
Oct 30 20:22:17.766863 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:17.766763    5958 cpu_manager.go:414] "containerMap: removing container" podUID="30153d8d-0b5e-4d18-a442-c82144050a85" containerName="revoke-keys"
Oct 30 20:22:17.766863 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:17.766768    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30153d8d-0b5e-4d18-a442-c82144050a85" containerName="revoke-keys"
Oct 30 20:22:17.766863 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:17.766777    5958 cpu_manager.go:414] "containerMap: removing container" podUID="6bc60e1a-391d-43b0-8b8e-2f6f33c234be" containerName="istio-proxy"
Oct 30 20:22:17.766863 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:17.766785    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc60e1a-391d-43b0-8b8e-2f6f33c234be" containerName="istio-proxy"
Oct 30 20:22:17.767041 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:17.766911    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc60e1a-391d-43b0-8b8e-2f6f33c234be" containerName="istio-proxy"
Oct 30 20:22:17.767041 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:17.766922    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="30153d8d-0b5e-4d18-a442-c82144050a85" containerName="revoke-keys"
Oct 30 20:22:17.771941 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:17.771916    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986"
Oct 30 20:22:17.827182 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:17.827147    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986"]
Oct 30 20:22:17.879475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:17.874711    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-14c846-dockercfg-5gsdq\""
Oct 30 20:22:17.950872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:17.932927    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvq8z\" (UniqueName: \"kubernetes.io/projected/ee6329a5-2f0b-47c7-84ec-25d1c72ba396-kube-api-access-dvq8z\") pod \"payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986\" (UID: \"ee6329a5-2f0b-47c7-84ec-25d1c72ba396\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986"
Oct 30 20:22:17.950872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:17.933047    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee6329a5-2f0b-47c7-84ec-25d1c72ba396-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986\" (UID: \"ee6329a5-2f0b-47c7-84ec-25d1c72ba396\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986"
Oct 30 20:22:17.950872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:17.933416    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-14c846\""
Oct 30 20:22:17.950872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:17.936351    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6"]
Oct 30 20:22:17.950872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:17.937560    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc60e1a-391d-43b0-8b8e-2f6f33c234be" containerName="istio-proxy"
Oct 30 20:22:17.950872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:17.950728    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6"
Oct 30 20:22:17.977427 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:17.977393    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6"]
Oct 30 20:22:18.034655 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:18.034575    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvq8z\" (UniqueName: \"kubernetes.io/projected/ee6329a5-2f0b-47c7-84ec-25d1c72ba396-kube-api-access-dvq8z\") pod \"payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986\" (UID: \"ee6329a5-2f0b-47c7-84ec-25d1c72ba396\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986"
Oct 30 20:22:18.034802 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:18.034656    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee6329a5-2f0b-47c7-84ec-25d1c72ba396-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986\" (UID: \"ee6329a5-2f0b-47c7-84ec-25d1c72ba396\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986"
Oct 30 20:22:18.035330 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:18.035304    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee6329a5-2f0b-47c7-84ec-25d1c72ba396-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986\" (UID: \"ee6329a5-2f0b-47c7-84ec-25d1c72ba396\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986"
Oct 30 20:22:18.096282 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:18.096250    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvq8z\" (UniqueName: \"kubernetes.io/projected/ee6329a5-2f0b-47c7-84ec-25d1c72ba396-kube-api-access-dvq8z\") pod \"payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986\" (UID: \"ee6329a5-2f0b-47c7-84ec-25d1c72ba396\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986"
Oct 30 20:22:18.136150 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:18.136108    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvh5l\" (UniqueName: \"kubernetes.io/projected/2773ad9c-dde9-409c-937d-2a85268696f1-kube-api-access-tvh5l\") pod \"payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6\" (UID: \"2773ad9c-dde9-409c-937d-2a85268696f1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6"
Oct 30 20:22:18.136342 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:18.136187    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2773ad9c-dde9-409c-937d-2a85268696f1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6\" (UID: \"2773ad9c-dde9-409c-937d-2a85268696f1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6"
Oct 30 20:22:18.237420 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:18.237384    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvh5l\" (UniqueName: \"kubernetes.io/projected/2773ad9c-dde9-409c-937d-2a85268696f1-kube-api-access-tvh5l\") pod \"payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6\" (UID: \"2773ad9c-dde9-409c-937d-2a85268696f1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6"
Oct 30 20:22:18.237420 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:18.237421    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2773ad9c-dde9-409c-937d-2a85268696f1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6\" (UID: \"2773ad9c-dde9-409c-937d-2a85268696f1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6"
Oct 30 20:22:18.238189 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:18.238168    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2773ad9c-dde9-409c-937d-2a85268696f1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6\" (UID: \"2773ad9c-dde9-409c-937d-2a85268696f1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6"
Oct 30 20:22:18.298718 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:18.298649    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvh5l\" (UniqueName: \"kubernetes.io/projected/2773ad9c-dde9-409c-937d-2a85268696f1-kube-api-access-tvh5l\") pod \"payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6\" (UID: \"2773ad9c-dde9-409c-937d-2a85268696f1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6"
Oct 30 20:22:18.386100 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:18.386066    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986"
Oct 30 20:22:18.573246 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:18.573218    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6"
Oct 30 20:22:18.773380 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:18.772695    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986"]
Oct 30 20:22:18.775767 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:22:18.775736    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee6329a5_2f0b_47c7_84ec_25d1c72ba396.slice/crio-dc1fdc1b5453870429a4ebb66ce8ac1b60def463b325742c300b0b54f6aba95f WatchSource:0}: Error finding container dc1fdc1b5453870429a4ebb66ce8ac1b60def463b325742c300b0b54f6aba95f: Status 404 returned error can't find the container with id dc1fdc1b5453870429a4ebb66ce8ac1b60def463b325742c300b0b54f6aba95f
Oct 30 20:22:18.977476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:18.967058    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6"]
Oct 30 20:22:18.977476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:18.972508    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986" event={"ID":"ee6329a5-2f0b-47c7-84ec-25d1c72ba396","Type":"ContainerStarted","Data":"5e9aab54d9a95dd3a734df96406946af48a01179ad8629fa21fab3ebc656e95d"}
Oct 30 20:22:18.977476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:18.972602    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986"
Oct 30 20:22:18.977476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:18.972679    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986" event={"ID":"ee6329a5-2f0b-47c7-84ec-25d1c72ba396","Type":"ContainerStarted","Data":"dc1fdc1b5453870429a4ebb66ce8ac1b60def463b325742c300b0b54f6aba95f"}
Oct 30 20:22:18.984546 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:22:18.984516    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2773ad9c_dde9_409c_937d_2a85268696f1.slice/crio-2987548a5957e68603f6341798fb5d8894efaa21d21891681b6f12e8420f25f0 WatchSource:0}: Error finding container 2987548a5957e68603f6341798fb5d8894efaa21d21891681b6f12e8420f25f0: Status 404 returned error can't find the container with id 2987548a5957e68603f6341798fb5d8894efaa21d21891681b6f12e8420f25f0
Oct 30 20:22:19.980535 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:19.980493    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6" event={"ID":"2773ad9c-dde9-409c-937d-2a85268696f1","Type":"ContainerStarted","Data":"824047327c083de6da5d33c09ff11b0f9fa1aba65fb5b65a16b059097e4c4ba8"}
Oct 30 20:22:19.980982 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:19.980560    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6" event={"ID":"2773ad9c-dde9-409c-937d-2a85268696f1","Type":"ContainerStarted","Data":"2987548a5957e68603f6341798fb5d8894efaa21d21891681b6f12e8420f25f0"}
Oct 30 20:22:19.980982 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:19.980753    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6"
Oct 30 20:22:20.101289 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:20.101232    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986" podStartSLOduration=3.101217332 podStartE2EDuration="3.101217332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:19.094327143 +0000 UTC m=+2377.826542535" watchObservedRunningTime="2026-10-30 20:22:20.101217332 +0000 UTC m=+2378.833432721"
Oct 30 20:22:20.101505 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:20.101317    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6" podStartSLOduration=3.101314176 podStartE2EDuration="3.101314176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:20.097993184 +0000 UTC m=+2378.830208574" watchObservedRunningTime="2026-10-30 20:22:20.101314176 +0000 UTC m=+2378.833529565"
Oct 30 20:22:29.983508 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:29.983478    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986"
Oct 30 20:22:31.000102 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:31.000067    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6"
Oct 30 20:22:36.606706 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:22:36.606660    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:22:41.828969 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:41.828929    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 20:22:41.836203 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:41.836177    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 20:22:42.369372 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:42.369342    5958 scope.go:122] "RemoveContainer" containerID="80855c98ca95a234482722c9e14a075988a01af2bab1918f541d3a01b0b8e254"
Oct 30 20:22:52.812725 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:52.812691    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx"]
Oct 30 20:22:52.818769 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:52.818747    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx"
Oct 30 20:22:52.871731 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:52.871697    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-453277\""
Oct 30 20:22:52.924580 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:52.924491    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx"]
Oct 30 20:22:53.013554 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:53.013417    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krlbr\" (UniqueName: \"kubernetes.io/projected/c9fa71b4-8a5b-40e6-be72-35583f6560f5-kube-api-access-krlbr\") pod \"payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx\" (UID: \"c9fa71b4-8a5b-40e6-be72-35583f6560f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx"
Oct 30 20:22:53.014065 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:53.014038    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9fa71b4-8a5b-40e6-be72-35583f6560f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx\" (UID: \"c9fa71b4-8a5b-40e6-be72-35583f6560f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx"
Oct 30 20:22:53.016581 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:53.016550    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm"]
Oct 30 20:22:53.030517 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:53.028893    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm"
Oct 30 20:22:53.043471 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:53.043066    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm"]
Oct 30 20:22:53.084517 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:53.084490    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-453277-dockercfg-z7lvp\""
Oct 30 20:22:53.116181 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:53.116144    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krlbr\" (UniqueName: \"kubernetes.io/projected/c9fa71b4-8a5b-40e6-be72-35583f6560f5-kube-api-access-krlbr\") pod \"payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx\" (UID: \"c9fa71b4-8a5b-40e6-be72-35583f6560f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx"
Oct 30 20:22:53.116364 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:53.116278    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9fa71b4-8a5b-40e6-be72-35583f6560f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx\" (UID: \"c9fa71b4-8a5b-40e6-be72-35583f6560f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx"
Oct 30 20:22:53.116997 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:53.116978    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9fa71b4-8a5b-40e6-be72-35583f6560f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx\" (UID: \"c9fa71b4-8a5b-40e6-be72-35583f6560f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx"
Oct 30 20:22:53.177824 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:53.177786    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krlbr\" (UniqueName: \"kubernetes.io/projected/c9fa71b4-8a5b-40e6-be72-35583f6560f5-kube-api-access-krlbr\") pod \"payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx\" (UID: \"c9fa71b4-8a5b-40e6-be72-35583f6560f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx"
Oct 30 20:22:53.217804 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:53.217767    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnfgf\" (UniqueName: \"kubernetes.io/projected/60327e5a-6b34-4eff-bfaa-6af3715d0b78-kube-api-access-pnfgf\") pod \"payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm\" (UID: \"60327e5a-6b34-4eff-bfaa-6af3715d0b78\") " pod="openshift-ingress/payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm"
Oct 30 20:22:53.218034 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:53.218012    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60327e5a-6b34-4eff-bfaa-6af3715d0b78-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm\" (UID: \"60327e5a-6b34-4eff-bfaa-6af3715d0b78\") " pod="openshift-ingress/payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm"
Oct 30 20:22:53.319524 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:53.319487    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnfgf\" (UniqueName: \"kubernetes.io/projected/60327e5a-6b34-4eff-bfaa-6af3715d0b78-kube-api-access-pnfgf\") pod \"payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm\" (UID: \"60327e5a-6b34-4eff-bfaa-6af3715d0b78\") " pod="openshift-ingress/payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm"
Oct 30 20:22:53.319786 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:53.319766    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60327e5a-6b34-4eff-bfaa-6af3715d0b78-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm\" (UID: \"60327e5a-6b34-4eff-bfaa-6af3715d0b78\") " pod="openshift-ingress/payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm"
Oct 30 20:22:53.320644 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:53.320619    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60327e5a-6b34-4eff-bfaa-6af3715d0b78-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm\" (UID: \"60327e5a-6b34-4eff-bfaa-6af3715d0b78\") " pod="openshift-ingress/payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm"
Oct 30 20:22:53.380992 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:53.380917    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnfgf\" (UniqueName: \"kubernetes.io/projected/60327e5a-6b34-4eff-bfaa-6af3715d0b78-kube-api-access-pnfgf\") pod \"payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm\" (UID: \"60327e5a-6b34-4eff-bfaa-6af3715d0b78\") " pod="openshift-ingress/payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm"
Oct 30 20:22:53.432161 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:53.432123    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx"
Oct 30 20:22:53.652384 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:53.652304    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm"
Oct 30 20:22:53.810782 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:53.810744    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx"]
Oct 30 20:22:53.812524 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:22:53.812499    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9fa71b4_8a5b_40e6_be72_35583f6560f5.slice/crio-cca15862347dd09055215355748657fde10a84faf32fcf2c710c1a18495db70e WatchSource:0}: Error finding container cca15862347dd09055215355748657fde10a84faf32fcf2c710c1a18495db70e: Status 404 returned error can't find the container with id cca15862347dd09055215355748657fde10a84faf32fcf2c710c1a18495db70e
Oct 30 20:22:54.033119 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:54.033092    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm"]
Oct 30 20:22:54.035252 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:22:54.035226    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60327e5a_6b34_4eff_bfaa_6af3715d0b78.slice/crio-bc43f18d02148ae0035c35e64d7973112f8f105192ef003ea148994057dc72c5 WatchSource:0}: Error finding container bc43f18d02148ae0035c35e64d7973112f8f105192ef003ea148994057dc72c5: Status 404 returned error can't find the container with id bc43f18d02148ae0035c35e64d7973112f8f105192ef003ea148994057dc72c5
Oct 30 20:22:54.222157 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:54.222116    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm" event={"ID":"60327e5a-6b34-4eff-bfaa-6af3715d0b78","Type":"ContainerStarted","Data":"16946d3da87ea5268252d3104a208d4928f430331822ee23e87210dd2e0964ff"}
Oct 30 20:22:54.222308 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:54.222181    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm" event={"ID":"60327e5a-6b34-4eff-bfaa-6af3715d0b78","Type":"ContainerStarted","Data":"bc43f18d02148ae0035c35e64d7973112f8f105192ef003ea148994057dc72c5"}
Oct 30 20:22:54.222308 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:54.222209    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm"
Oct 30 20:22:54.223850 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:54.223825    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx" event={"ID":"c9fa71b4-8a5b-40e6-be72-35583f6560f5","Type":"ContainerStarted","Data":"b6fbd421df27db7bb75bd7f6ed2cade3c2e2879354336cdb04fb46e4fe4e00ab"}
Oct 30 20:22:54.223992 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:54.223864    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx" event={"ID":"c9fa71b4-8a5b-40e6-be72-35583f6560f5","Type":"ContainerStarted","Data":"cca15862347dd09055215355748657fde10a84faf32fcf2c710c1a18495db70e"}
Oct 30 20:22:54.223992 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:54.223965    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx"
Oct 30 20:22:54.340480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:54.340399    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm" podStartSLOduration=2.340385515 podStartE2EDuration="2.340385515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:54.337593329 +0000 UTC m=+2413.069808720" watchObservedRunningTime="2026-10-30 20:22:54.340385515 +0000 UTC m=+2413.072600905"
Oct 30 20:22:54.391059 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:54.391018    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"]
Oct 30 20:22:54.391231 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:54.391192    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x" podUID="df7304e3-998f-4610-acf5-aee21d0ee2ad" containerName="istio-proxy" containerID="cri-o://5d8d7c0292f29948cbce12a9ae37ee3d98de51b756739d9062d09c47afef39a8" gracePeriod=30
Oct 30 20:22:54.453845 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:54.453784    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx" podStartSLOduration=2.453764815 podStartE2EDuration="2.453764815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:54.453743052 +0000 UTC m=+2413.185958445" watchObservedRunningTime="2026-10-30 20:22:54.453764815 +0000 UTC m=+2413.185980206"
Oct 30 20:22:57.061185 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:57.061139    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:22:57.061619 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:57.061212    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x" podUID="df7304e3-998f-4610-acf5-aee21d0ee2ad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:22:59.675882 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.675809    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:22:59.712653 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.712611    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-workload-certs\") pod \"df7304e3-998f-4610-acf5-aee21d0ee2ad\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") "
Oct 30 20:22:59.712857 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.712698    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-podinfo\") pod \"df7304e3-998f-4610-acf5-aee21d0ee2ad\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") "
Oct 30 20:22:59.712857 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.712768    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-credential-socket\") pod \"df7304e3-998f-4610-acf5-aee21d0ee2ad\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") "
Oct 30 20:22:59.712857 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.712798    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-token\" (UniqueName: \"kubernetes.io/projected/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-token\") pod \"df7304e3-998f-4610-acf5-aee21d0ee2ad\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") "
Oct 30 20:22:59.713015 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.712925    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-workload-certs" pod "df7304e3-998f-4610-acf5-aee21d0ee2ad" (UID: "df7304e3-998f-4610-acf5-aee21d0ee2ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:59.713015 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.712936    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-envoy\") pod \"df7304e3-998f-4610-acf5-aee21d0ee2ad\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") "
Oct 30 20:22:59.713120 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.713075    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-credential-socket" pod "df7304e3-998f-4610-acf5-aee21d0ee2ad" (UID: "df7304e3-998f-4610-acf5-aee21d0ee2ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:59.713176 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.713137    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df7304e3-998f-4610-acf5-aee21d0ee2ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df7304e3-998f-4610-acf5-aee21d0ee2ad-istiod-ca-cert\") pod \"df7304e3-998f-4610-acf5-aee21d0ee2ad\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") "
Oct 30 20:22:59.713176 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.713163    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df7304e3-998f-4610-acf5-aee21d0ee2ad-kube-api-access-44hrn\" (UniqueName: \"kubernetes.io/projected/df7304e3-998f-4610-acf5-aee21d0ee2ad-kube-api-access-44hrn\") pod \"df7304e3-998f-4610-acf5-aee21d0ee2ad\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") "
Oct 30 20:22:59.713272 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.713202    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-data\") pod \"df7304e3-998f-4610-acf5-aee21d0ee2ad\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") "
Oct 30 20:22:59.713272 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.713236    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-workload-socket\") pod \"df7304e3-998f-4610-acf5-aee21d0ee2ad\" (UID: \"df7304e3-998f-4610-acf5-aee21d0ee2ad\") "
Oct 30 20:22:59.713583 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.713537    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-workload-socket" pod "df7304e3-998f-4610-acf5-aee21d0ee2ad" (UID: "df7304e3-998f-4610-acf5-aee21d0ee2ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:59.713583 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.713553    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df7304e3-998f-4610-acf5-aee21d0ee2ad-istiod-ca-cert" pod "df7304e3-998f-4610-acf5-aee21d0ee2ad" (UID: "df7304e3-998f-4610-acf5-aee21d0ee2ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:59.713951 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.713577    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-data" pod "df7304e3-998f-4610-acf5-aee21d0ee2ad" (UID: "df7304e3-998f-4610-acf5-aee21d0ee2ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:59.713951 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.713936    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:59.714076 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.713956    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:59.714076 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.713973    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df7304e3-998f-4610-acf5-aee21d0ee2ad-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:59.714076 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.713987    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:59.714076 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.714000    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:59.715353 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.715317    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-podinfo" pod "df7304e3-998f-4610-acf5-aee21d0ee2ad" (UID: "df7304e3-998f-4610-acf5-aee21d0ee2ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:22:59.715492 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.715382    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-token" pod "df7304e3-998f-4610-acf5-aee21d0ee2ad" (UID: "df7304e3-998f-4610-acf5-aee21d0ee2ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:59.715492 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.715399    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-envoy" pod "df7304e3-998f-4610-acf5-aee21d0ee2ad" (UID: "df7304e3-998f-4610-acf5-aee21d0ee2ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:59.716029 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.716010    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df7304e3-998f-4610-acf5-aee21d0ee2ad-kube-api-access-44hrn" pod "df7304e3-998f-4610-acf5-aee21d0ee2ad" (UID: "df7304e3-998f-4610-acf5-aee21d0ee2ad"). InnerVolumeSpecName "kube-api-access-44hrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:59.816011 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.815974    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44hrn\" (UniqueName: \"kubernetes.io/projected/df7304e3-998f-4610-acf5-aee21d0ee2ad-kube-api-access-44hrn\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:59.816011 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.816009    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:59.816011 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.816020    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:59.816011 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:22:59.816029    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df7304e3-998f-4610-acf5-aee21d0ee2ad-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:00.270033 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:00.269999    5958 generic.go:356] "Generic (PLEG): container finished" podID="df7304e3-998f-4610-acf5-aee21d0ee2ad" containerID="5d8d7c0292f29948cbce12a9ae37ee3d98de51b756739d9062d09c47afef39a8" exitCode=0
Oct 30 20:23:00.270193 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:00.270045    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x" event={"ID":"df7304e3-998f-4610-acf5-aee21d0ee2ad","Type":"ContainerDied","Data":"5d8d7c0292f29948cbce12a9ae37ee3d98de51b756739d9062d09c47afef39a8"}
Oct 30 20:23:00.270193 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:00.270052    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"
Oct 30 20:23:00.270193 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:00.270087    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x" event={"ID":"df7304e3-998f-4610-acf5-aee21d0ee2ad","Type":"ContainerDied","Data":"8f873a8f39deb6fdddb74a2a283a0f713b863bc195426f31fae6bbd7cf9e7d5c"}
Oct 30 20:23:00.270193 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:00.270109    5958 scope.go:122] "RemoveContainer" containerID="5d8d7c0292f29948cbce12a9ae37ee3d98de51b756739d9062d09c47afef39a8"
Oct 30 20:23:00.281082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:00.281057    5958 scope.go:122] "RemoveContainer" containerID="5d8d7c0292f29948cbce12a9ae37ee3d98de51b756739d9062d09c47afef39a8"
Oct 30 20:23:00.281376 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:23:00.281357    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d8d7c0292f29948cbce12a9ae37ee3d98de51b756739d9062d09c47afef39a8\": container with ID starting with 5d8d7c0292f29948cbce12a9ae37ee3d98de51b756739d9062d09c47afef39a8 not found: ID does not exist" containerID="5d8d7c0292f29948cbce12a9ae37ee3d98de51b756739d9062d09c47afef39a8"
Oct 30 20:23:00.281413 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:00.281385    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d8d7c0292f29948cbce12a9ae37ee3d98de51b756739d9062d09c47afef39a8"} err="failed to get container status \"5d8d7c0292f29948cbce12a9ae37ee3d98de51b756739d9062d09c47afef39a8\": rpc error: code = NotFound desc = could not find container \"5d8d7c0292f29948cbce12a9ae37ee3d98de51b756739d9062d09c47afef39a8\": container with ID starting with 5d8d7c0292f29948cbce12a9ae37ee3d98de51b756739d9062d09c47afef39a8 not found: ID does not exist"
Oct 30 20:23:00.437293 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:00.437259    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"]
Oct 30 20:23:00.487309 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:00.487270    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-xq89x"]
Oct 30 20:23:01.398843 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:01.398807    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df7304e3-998f-4610-acf5-aee21d0ee2ad" path="/var/lib/kubelet/pods/df7304e3-998f-4610-acf5-aee21d0ee2ad/volumes"
Oct 30 20:23:05.233846 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:05.233815    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm"
Oct 30 20:23:05.234219 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:05.233883    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx"
Oct 30 20:23:06.606601 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:23:06.606565    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:23:11.353777 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:11.353744    5958 generic.go:356] "Generic (PLEG): container finished" podID="04b5051c-1bff-4550-b442-7f0b7622a271" containerID="265b226f7c326898944e7502af9464b56f87ce3e1929059ef9f48424883a29b9" exitCode=0
Oct 30 20:23:11.354214 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:11.353828    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7" event={"ID":"04b5051c-1bff-4550-b442-7f0b7622a271","Type":"ContainerDied","Data":"265b226f7c326898944e7502af9464b56f87ce3e1929059ef9f48424883a29b9"}
Oct 30 20:23:11.354287 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:11.354264    5958 scope.go:122] "RemoveContainer" containerID="265b226f7c326898944e7502af9464b56f87ce3e1929059ef9f48424883a29b9"
Oct 30 20:23:11.355762 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:11.355729    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:23:11.355870 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:11.355808    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:23:11.355939 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:11.355888    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:23:12.367863 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:12.367827    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7" event={"ID":"04b5051c-1bff-4550-b442-7f0b7622a271","Type":"ContainerStarted","Data":"2c09c75f16955668a6ec949c6edc350888d135419a38e7d621f19360b8c44a1a"}
Oct 30 20:23:13.358971 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:13.358839    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:23:13.361816 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:13.361788    5958 patch_prober.go:29] interesting pod/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 20:23:13.361966 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:13.361834    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7" podUID="04b5051c-1bff-4550-b442-7f0b7622a271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 20:23:14.360506 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:14.359560    5958 patch_prober.go:29] interesting pod/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 20:23:14.360506 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:14.359630    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7" podUID="04b5051c-1bff-4550-b442-7f0b7622a271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 20:23:15.156477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:15.154539    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:23:15.360595 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:15.360543    5958 patch_prober.go:29] interesting pod/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 20:23:15.360595 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:15.360594    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7" podUID="04b5051c-1bff-4550-b442-7f0b7622a271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 20:23:16.355480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:16.355148    5958 patch_prober.go:29] interesting pod/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 20:23:16.355480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:16.355205    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7" podUID="04b5051c-1bff-4550-b442-7f0b7622a271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 20:23:17.025013 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.024977    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"]
Oct 30 20:23:17.025728 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.025709    5958 cpu_manager.go:414] "containerMap: removing container" podUID="df7304e3-998f-4610-acf5-aee21d0ee2ad" containerName="istio-proxy"
Oct 30 20:23:17.025791 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.025729    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df7304e3-998f-4610-acf5-aee21d0ee2ad" containerName="istio-proxy"
Oct 30 20:23:17.025843 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.025833    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="df7304e3-998f-4610-acf5-aee21d0ee2ad" containerName="istio-proxy"
Oct 30 20:23:17.028848 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.028829    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:23:17.112763 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.112712    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-model-cache\") pod \"test-model-0051fb-kserve-75f97765b7-gxcxq\" (UID: \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\") " pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:23:17.112949 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.112784    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f0b2bb-c5b0-4603-a096-09247fc67aa3-tls-certs\") pod \"test-model-0051fb-kserve-75f97765b7-gxcxq\" (UID: \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\") " pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:23:17.113020 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.112970    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-dshm\") pod \"test-model-0051fb-kserve-75f97765b7-gxcxq\" (UID: \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\") " pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:23:17.113081 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.113041    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mscm8\" (UniqueName: \"kubernetes.io/projected/52f0b2bb-c5b0-4603-a096-09247fc67aa3-kube-api-access-mscm8\") pod \"test-model-0051fb-kserve-75f97765b7-gxcxq\" (UID: \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\") " pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:23:17.113210 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.113174    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-home\") pod \"test-model-0051fb-kserve-75f97765b7-gxcxq\" (UID: \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\") " pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:23:17.113350 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.113328    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-tmp-dir\") pod \"test-model-0051fb-kserve-75f97765b7-gxcxq\" (UID: \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\") " pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:23:17.134765 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.134683    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0051fb\"/\"default-dockercfg-9ljpm\""
Oct 30 20:23:17.134911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.134836    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0051fb\"/\"test-model-0051fb-kserve-self-signed-certs\""
Oct 30 20:23:17.134911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.134898    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0051fb\"/\"openshift-service-ca.crt\""
Oct 30 20:23:17.135029 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.134913    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0051fb\"/\"kube-root-ca.crt\""
Oct 30 20:23:17.145310 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.145277    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"]
Oct 30 20:23:17.215174 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.215133    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-tmp-dir\") pod \"test-model-0051fb-kserve-75f97765b7-gxcxq\" (UID: \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\") " pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:23:17.215400 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.215283    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-model-cache\") pod \"test-model-0051fb-kserve-75f97765b7-gxcxq\" (UID: \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\") " pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:23:17.215400 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.215338    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f0b2bb-c5b0-4603-a096-09247fc67aa3-tls-certs\") pod \"test-model-0051fb-kserve-75f97765b7-gxcxq\" (UID: \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\") " pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:23:17.215561 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.215465    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-dshm\") pod \"test-model-0051fb-kserve-75f97765b7-gxcxq\" (UID: \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\") " pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:23:17.215561 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.215536    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mscm8\" (UniqueName: \"kubernetes.io/projected/52f0b2bb-c5b0-4603-a096-09247fc67aa3-kube-api-access-mscm8\") pod \"test-model-0051fb-kserve-75f97765b7-gxcxq\" (UID: \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\") " pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:23:17.215676 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.215596    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-home\") pod \"test-model-0051fb-kserve-75f97765b7-gxcxq\" (UID: \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\") " pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:23:17.216160 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.216102    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-home\") pod \"test-model-0051fb-kserve-75f97765b7-gxcxq\" (UID: \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\") " pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:23:17.216408 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.216383    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-tmp-dir\") pod \"test-model-0051fb-kserve-75f97765b7-gxcxq\" (UID: \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\") " pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:23:17.216824 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.216799    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-model-cache\") pod \"test-model-0051fb-kserve-75f97765b7-gxcxq\" (UID: \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\") " pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:23:17.219286 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.219262    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-dshm\") pod \"test-model-0051fb-kserve-75f97765b7-gxcxq\" (UID: \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\") " pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:23:17.219405 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.219300    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f0b2bb-c5b0-4603-a096-09247fc67aa3-tls-certs\") pod \"test-model-0051fb-kserve-75f97765b7-gxcxq\" (UID: \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\") " pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:23:17.278283 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.278208    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mscm8\" (UniqueName: \"kubernetes.io/projected/52f0b2bb-c5b0-4603-a096-09247fc67aa3-kube-api-access-mscm8\") pod \"test-model-0051fb-kserve-75f97765b7-gxcxq\" (UID: \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\") " pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:23:17.355230 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.355194    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:23:17.385354 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.385322    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:23:17.387376 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.387339    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:23:17.779959 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.779924    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"]
Oct 30 20:23:17.780181 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.780152    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw" podUID="56ec387e-8e34-472e-b8f5-032a57c517e9" containerName="istio-proxy" containerID="cri-o://7036812e0caf371d6f0c015abbfc0c9efd36138d7ed80f62d938d577c2ce2b03" gracePeriod=30
Oct 30 20:23:17.788084 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:17.788050    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"]
Oct 30 20:23:17.796408 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:23:17.796378    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52f0b2bb_c5b0_4603_a096_09247fc67aa3.slice/crio-ccc932ef62218789908201f5f2e8ff20aac7de50a820f6d9589fad6cf13cdd41 WatchSource:0}: Error finding container ccc932ef62218789908201f5f2e8ff20aac7de50a820f6d9589fad6cf13cdd41: Status 404 returned error can't find the container with id ccc932ef62218789908201f5f2e8ff20aac7de50a820f6d9589fad6cf13cdd41
Oct 30 20:23:18.413672 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:18.413641    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq" event={"ID":"52f0b2bb-c5b0-4603-a096-09247fc67aa3","Type":"ContainerStarted","Data":"d442dd6b142cd70efe3b263b42b973185ace90880888e9286d47d19a3b07663e"}
Oct 30 20:23:18.413672 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:18.413678    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq" event={"ID":"52f0b2bb-c5b0-4603-a096-09247fc67aa3","Type":"ContainerStarted","Data":"ccc932ef62218789908201f5f2e8ff20aac7de50a820f6d9589fad6cf13cdd41"}
Oct 30 20:23:18.534445 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:18.534398    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq" podStartSLOduration=2.534384378 podStartE2EDuration="2.534384378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:18.532330499 +0000 UTC m=+2437.264545889" watchObservedRunningTime="2026-10-30 20:23:18.534384378 +0000 UTC m=+2437.266599958"
Oct 30 20:23:19.356148 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:19.356113    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:23:19.569299 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:19.569264    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"]
Oct 30 20:23:19.619481 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:19.619398    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"]
Oct 30 20:23:20.722747 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:20.722714    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"]
Oct 30 20:23:20.723248 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:20.722924    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9" podUID="68f462dc-feb6-44af-a8af-bc45ce2e0e77" containerName="istio-proxy" containerID="cri-o://4f904325479799b0c3a1d8f710a7c52f4d4aeab32fda4ddf3467a42a622eaf6f" gracePeriod=30
Oct 30 20:23:20.760638 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:20.760609    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"]
Oct 30 20:23:20.760863 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:20.760840    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m" podUID="a993fc51-49ae-4c3d-8c26-1f256e5064ed" containerName="istio-proxy" containerID="cri-o://2cdeb0eda1225c96d3d10d62205122c338b46fefce991375cd1310195fba8403" gracePeriod=30
Oct 30 20:23:20.764530 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:20.764506    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"]
Oct 30 20:23:20.764684 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:20.764666    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll" podUID="36638d21-d7dd-40ec-9a33-83dcbbe2d7ae" containerName="istio-proxy" containerID="cri-o://d67212398c218a75ab46f08d283ed2b36dc8ca00a5d93f34cbd805cc14d10c16" gracePeriod=30
Oct 30 20:23:20.775708 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:20.775670    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"]
Oct 30 20:23:20.775865 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:20.775840    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh" podUID="5aeb303b-220a-4b83-acb6-6c1004a977d1" containerName="istio-proxy" containerID="cri-o://c7c9950a63166d7b3f63c8eda06afef9617a8361f10e0975bef3bba17089127f" gracePeriod=30
Oct 30 20:23:21.897752 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:21.897712    5958 patch_prober.go:29] interesting pod/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:23:21.898162 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:21.897781    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh" podUID="5aeb303b-220a-4b83-acb6-6c1004a977d1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:23:23.063947 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.063924    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 20:23:23.211723 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.211693    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56ec387e-8e34-472e-b8f5-032a57c517e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56ec387e-8e34-472e-b8f5-032a57c517e9-istiod-ca-cert\") pod \"56ec387e-8e34-472e-b8f5-032a57c517e9\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") "
Oct 30 20:23:23.211926 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.211822    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-workload-socket\") pod \"56ec387e-8e34-472e-b8f5-032a57c517e9\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") "
Oct 30 20:23:23.211926 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.211847    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-credential-socket\") pod \"56ec387e-8e34-472e-b8f5-032a57c517e9\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") "
Oct 30 20:23:23.211926 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.211865    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-podinfo\") pod \"56ec387e-8e34-472e-b8f5-032a57c517e9\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") "
Oct 30 20:23:23.211926 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.211893    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-envoy\") pod \"56ec387e-8e34-472e-b8f5-032a57c517e9\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") "
Oct 30 20:23:23.212133 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.211997    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-data\") pod \"56ec387e-8e34-472e-b8f5-032a57c517e9\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") "
Oct 30 20:23:23.212133 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.212037    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-token\" (UniqueName: \"kubernetes.io/projected/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-token\") pod \"56ec387e-8e34-472e-b8f5-032a57c517e9\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") "
Oct 30 20:23:23.212133 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.212091    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-workload-certs\") pod \"56ec387e-8e34-472e-b8f5-032a57c517e9\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") "
Oct 30 20:23:23.212282 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.212163    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56ec387e-8e34-472e-b8f5-032a57c517e9-kube-api-access-wxz5c\" (UniqueName: \"kubernetes.io/projected/56ec387e-8e34-472e-b8f5-032a57c517e9-kube-api-access-wxz5c\") pod \"56ec387e-8e34-472e-b8f5-032a57c517e9\" (UID: \"56ec387e-8e34-472e-b8f5-032a57c517e9\") "
Oct 30 20:23:23.212282 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.212085    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ec387e-8e34-472e-b8f5-032a57c517e9-istiod-ca-cert" pod "56ec387e-8e34-472e-b8f5-032a57c517e9" (UID: "56ec387e-8e34-472e-b8f5-032a57c517e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:23.212282 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.212174    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-credential-socket" pod "56ec387e-8e34-472e-b8f5-032a57c517e9" (UID: "56ec387e-8e34-472e-b8f5-032a57c517e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:23.212432 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.212326    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-workload-socket" pod "56ec387e-8e34-472e-b8f5-032a57c517e9" (UID: "56ec387e-8e34-472e-b8f5-032a57c517e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:23.212432 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.212302    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-data" pod "56ec387e-8e34-472e-b8f5-032a57c517e9" (UID: "56ec387e-8e34-472e-b8f5-032a57c517e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:23.212805 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.212649    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-workload-certs" pod "56ec387e-8e34-472e-b8f5-032a57c517e9" (UID: "56ec387e-8e34-472e-b8f5-032a57c517e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:23.212941 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.212862    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:23.212941 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.212884    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:23.212941 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.212898    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56ec387e-8e34-472e-b8f5-032a57c517e9-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:23.212941 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.212907    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:23.212941 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.212915    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:23.214372 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.214351    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-podinfo" pod "56ec387e-8e34-472e-b8f5-032a57c517e9" (UID: "56ec387e-8e34-472e-b8f5-032a57c517e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:23:23.214818 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.214786    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ec387e-8e34-472e-b8f5-032a57c517e9-kube-api-access-wxz5c" pod "56ec387e-8e34-472e-b8f5-032a57c517e9" (UID: "56ec387e-8e34-472e-b8f5-032a57c517e9"). InnerVolumeSpecName "kube-api-access-wxz5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:23.214903 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.214846    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-envoy" pod "56ec387e-8e34-472e-b8f5-032a57c517e9" (UID: "56ec387e-8e34-472e-b8f5-032a57c517e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:23.214944 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.214901    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-token" pod "56ec387e-8e34-472e-b8f5-032a57c517e9" (UID: "56ec387e-8e34-472e-b8f5-032a57c517e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:23.314656 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.314566    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:23.314656 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.314598    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxz5c\" (UniqueName: \"kubernetes.io/projected/56ec387e-8e34-472e-b8f5-032a57c517e9-kube-api-access-wxz5c\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:23.314656 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.314610    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:23.314656 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.314618    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56ec387e-8e34-472e-b8f5-032a57c517e9-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:23.447203 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.447164    5958 generic.go:356] "Generic (PLEG): container finished" podID="56ec387e-8e34-472e-b8f5-032a57c517e9" containerID="7036812e0caf371d6f0c015abbfc0c9efd36138d7ed80f62d938d577c2ce2b03" exitCode=0
Oct 30 20:23:23.447387 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.447224    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"
Oct 30 20:23:23.447387 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.447243    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw" event={"ID":"56ec387e-8e34-472e-b8f5-032a57c517e9","Type":"ContainerDied","Data":"7036812e0caf371d6f0c015abbfc0c9efd36138d7ed80f62d938d577c2ce2b03"}
Oct 30 20:23:23.447387 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.447298    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw" event={"ID":"56ec387e-8e34-472e-b8f5-032a57c517e9","Type":"ContainerDied","Data":"9abb42f96a7d49aa41cb7bb2533a8cd17d390f7acfc73341d0316e0fa4cc8556"}
Oct 30 20:23:23.447387 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.447330    5958 scope.go:122] "RemoveContainer" containerID="7036812e0caf371d6f0c015abbfc0c9efd36138d7ed80f62d938d577c2ce2b03"
Oct 30 20:23:23.457509 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.457488    5958 scope.go:122] "RemoveContainer" containerID="a706cb79798838937a181d0a6787ccfddc79c46487fd13d6073c0ea1910bba0c"
Oct 30 20:23:23.467113 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.467093    5958 scope.go:122] "RemoveContainer" containerID="7036812e0caf371d6f0c015abbfc0c9efd36138d7ed80f62d938d577c2ce2b03"
Oct 30 20:23:23.467396 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:23:23.467376    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7036812e0caf371d6f0c015abbfc0c9efd36138d7ed80f62d938d577c2ce2b03\": container with ID starting with 7036812e0caf371d6f0c015abbfc0c9efd36138d7ed80f62d938d577c2ce2b03 not found: ID does not exist" containerID="7036812e0caf371d6f0c015abbfc0c9efd36138d7ed80f62d938d577c2ce2b03"
Oct 30 20:23:23.467533 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.467406    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7036812e0caf371d6f0c015abbfc0c9efd36138d7ed80f62d938d577c2ce2b03"} err="failed to get container status \"7036812e0caf371d6f0c015abbfc0c9efd36138d7ed80f62d938d577c2ce2b03\": rpc error: code = NotFound desc = could not find container \"7036812e0caf371d6f0c015abbfc0c9efd36138d7ed80f62d938d577c2ce2b03\": container with ID starting with 7036812e0caf371d6f0c015abbfc0c9efd36138d7ed80f62d938d577c2ce2b03 not found: ID does not exist"
Oct 30 20:23:23.467533 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.467425    5958 scope.go:122] "RemoveContainer" containerID="a706cb79798838937a181d0a6787ccfddc79c46487fd13d6073c0ea1910bba0c"
Oct 30 20:23:23.467687 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:23:23.467669    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a706cb79798838937a181d0a6787ccfddc79c46487fd13d6073c0ea1910bba0c\": container with ID starting with a706cb79798838937a181d0a6787ccfddc79c46487fd13d6073c0ea1910bba0c not found: ID does not exist" containerID="a706cb79798838937a181d0a6787ccfddc79c46487fd13d6073c0ea1910bba0c"
Oct 30 20:23:23.467741 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.467697    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a706cb79798838937a181d0a6787ccfddc79c46487fd13d6073c0ea1910bba0c"} err="failed to get container status \"a706cb79798838937a181d0a6787ccfddc79c46487fd13d6073c0ea1910bba0c\": rpc error: code = NotFound desc = could not find container \"a706cb79798838937a181d0a6787ccfddc79c46487fd13d6073c0ea1910bba0c\": container with ID starting with a706cb79798838937a181d0a6787ccfddc79c46487fd13d6073c0ea1910bba0c not found: ID does not exist"
Oct 30 20:23:23.617255 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.617164    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"]
Oct 30 20:23:23.620058 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:23.620029    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d67ea9-openshift-default-77559f5d65-8t9jw"]
Oct 30 20:23:25.399777 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:25.399743    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56ec387e-8e34-472e-b8f5-032a57c517e9" path="/var/lib/kubelet/pods/56ec387e-8e34-472e-b8f5-032a57c517e9/volumes"
Oct 30 20:23:26.071960 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.071930    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:23:26.104585 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.104547    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:23:26.143332 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.143299    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:23:26.146377 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.146358    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:23:26.250149 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.250110    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f462dc-feb6-44af-a8af-bc45ce2e0e77-kube-api-access-9p24b\" (UniqueName: \"kubernetes.io/projected/68f462dc-feb6-44af-a8af-bc45ce2e0e77-kube-api-access-9p24b\") pod \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") "
Oct 30 20:23:26.250149 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.250147    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-credential-socket\") pod \"5aeb303b-220a-4b83-acb6-6c1004a977d1\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") "
Oct 30 20:23:26.250400 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.250194    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-workload-certs\") pod \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") "
Oct 30 20:23:26.250400 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.250219    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a993fc51-49ae-4c3d-8c26-1f256e5064ed-kube-api-access-v6jrh\" (UniqueName: \"kubernetes.io/projected/a993fc51-49ae-4c3d-8c26-1f256e5064ed-kube-api-access-v6jrh\") pod \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") "
Oct 30 20:23:26.250400 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.250247    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-credential-socket\") pod \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") "
Oct 30 20:23:26.250400 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.250275    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-credential-socket\") pod \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") "
Oct 30 20:23:26.250400 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.250310    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-envoy\") pod \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") "
Oct 30 20:23:26.250400 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.250344    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-workload-socket\") pod \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") "
Oct 30 20:23:26.250400 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.250371    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-envoy\") pod \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") "
Oct 30 20:23:26.250400 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.250400    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-workload-socket\") pod \"5aeb303b-220a-4b83-acb6-6c1004a977d1\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") "
Oct 30 20:23:26.250862 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.250432    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-podinfo\") pod \"5aeb303b-220a-4b83-acb6-6c1004a977d1\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") "
Oct 30 20:23:26.250862 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.250558    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-token\" (UniqueName: \"kubernetes.io/projected/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-token\") pod \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") "
Oct 30 20:23:26.250862 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.250592    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-podinfo\") pod \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") "
Oct 30 20:23:26.250862 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.250599    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-workload-certs" pod "36638d21-d7dd-40ec-9a33-83dcbbe2d7ae" (UID: "36638d21-d7dd-40ec-9a33-83dcbbe2d7ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.250862 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.250609    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-credential-socket" pod "68f462dc-feb6-44af-a8af-bc45ce2e0e77" (UID: "68f462dc-feb6-44af-a8af-bc45ce2e0e77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.250862 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.250672    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istiod-ca-cert\") pod \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") "
Oct 30 20:23:26.250862 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.250704    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-workload-socket\") pod \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") "
Oct 30 20:23:26.250862 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.250741    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-data\") pod \"5aeb303b-220a-4b83-acb6-6c1004a977d1\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") "
Oct 30 20:23:26.250862 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.250788    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-token\" (UniqueName: \"kubernetes.io/projected/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-token\") pod \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") "
Oct 30 20:23:26.250862 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.250854    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-workload-certs\") pod \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") "
Oct 30 20:23:26.251381 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.250887    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istiod-ca-cert\") pod \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") "
Oct 30 20:23:26.251381 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.250917    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-data\") pod \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") "
Oct 30 20:23:26.251381 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.250963    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-workload-certs\") pod \"5aeb303b-220a-4b83-acb6-6c1004a977d1\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") "
Oct 30 20:23:26.251381 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.251000    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-workload-socket\") pod \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") "
Oct 30 20:23:26.251381 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.251036    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-envoy\") pod \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") "
Oct 30 20:23:26.251381 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.251161    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-workload-socket" pod "5aeb303b-220a-4b83-acb6-6c1004a977d1" (UID: "5aeb303b-220a-4b83-acb6-6c1004a977d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.251381 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.251183    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-data\") pod \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") "
Oct 30 20:23:26.251381 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.251254    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-workload-socket" pod "a993fc51-49ae-4c3d-8c26-1f256e5064ed" (UID: "a993fc51-49ae-4c3d-8c26-1f256e5064ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.251381 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.251295    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-workload-certs\") pod \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") "
Oct 30 20:23:26.251381 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.251342    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-podinfo\") pod \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") "
Oct 30 20:23:26.251879 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.251395    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-token\" (UniqueName: \"kubernetes.io/projected/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-token\") pod \"5aeb303b-220a-4b83-acb6-6c1004a977d1\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") "
Oct 30 20:23:26.251879 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.251442    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aeb303b-220a-4b83-acb6-6c1004a977d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aeb303b-220a-4b83-acb6-6c1004a977d1-istiod-ca-cert\") pod \"5aeb303b-220a-4b83-acb6-6c1004a977d1\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") "
Oct 30 20:23:26.251879 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.251495    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-credential-socket\") pod \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") "
Oct 30 20:23:26.251879 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.251507    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-credential-socket" pod "36638d21-d7dd-40ec-9a33-83dcbbe2d7ae" (UID: "36638d21-d7dd-40ec-9a33-83dcbbe2d7ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.251879 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.251554    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-credential-socket" pod "5aeb303b-220a-4b83-acb6-6c1004a977d1" (UID: "5aeb303b-220a-4b83-acb6-6c1004a977d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.251879 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.251592    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aeb303b-220a-4b83-acb6-6c1004a977d1-kube-api-access-nljkw\" (UniqueName: \"kubernetes.io/projected/5aeb303b-220a-4b83-acb6-6c1004a977d1-kube-api-access-nljkw\") pod \"5aeb303b-220a-4b83-acb6-6c1004a977d1\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") "
Oct 30 20:23:26.251879 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.251634    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-kube-api-access-lmr6l\" (UniqueName: \"kubernetes.io/projected/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-kube-api-access-lmr6l\") pod \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") "
Oct 30 20:23:26.251879 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.251678    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-podinfo\") pod \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\" (UID: \"68f462dc-feb6-44af-a8af-bc45ce2e0e77\") "
Oct 30 20:23:26.251879 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.251708    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-envoy\") pod \"5aeb303b-220a-4b83-acb6-6c1004a977d1\" (UID: \"5aeb303b-220a-4b83-acb6-6c1004a977d1\") "
Oct 30 20:23:26.251879 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.251749    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istiod-ca-cert\") pod \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") "
Oct 30 20:23:26.251879 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.251783    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-data\") pod \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\" (UID: \"a993fc51-49ae-4c3d-8c26-1f256e5064ed\") "
Oct 30 20:23:26.251879 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.251822    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-token\" (UniqueName: \"kubernetes.io/projected/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-token\") pod \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\" (UID: \"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae\") "
Oct 30 20:23:26.252479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.251889    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istiod-ca-cert" pod "a993fc51-49ae-4c3d-8c26-1f256e5064ed" (UID: "a993fc51-49ae-4c3d-8c26-1f256e5064ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:26.252682 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.252602    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.252682 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.252630    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.252682 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.252645    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.252682 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.252659    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.252682 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.252674    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.252980 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.252687    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.252980 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.252700    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.252980 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.252893    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-envoy" pod "68f462dc-feb6-44af-a8af-bc45ce2e0e77" (UID: "68f462dc-feb6-44af-a8af-bc45ce2e0e77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.253723 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.253397    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f462dc-feb6-44af-a8af-bc45ce2e0e77-kube-api-access-9p24b" pod "68f462dc-feb6-44af-a8af-bc45ce2e0e77" (UID: "68f462dc-feb6-44af-a8af-bc45ce2e0e77"). InnerVolumeSpecName "kube-api-access-9p24b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:26.253723 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.253659    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-workload-socket" pod "68f462dc-feb6-44af-a8af-bc45ce2e0e77" (UID: "68f462dc-feb6-44af-a8af-bc45ce2e0e77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.253723 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.253665    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a993fc51-49ae-4c3d-8c26-1f256e5064ed-kube-api-access-v6jrh" pod "a993fc51-49ae-4c3d-8c26-1f256e5064ed" (UID: "a993fc51-49ae-4c3d-8c26-1f256e5064ed"). InnerVolumeSpecName "kube-api-access-v6jrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:26.253927 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.253860    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-data" pod "5aeb303b-220a-4b83-acb6-6c1004a977d1" (UID: "5aeb303b-220a-4b83-acb6-6c1004a977d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.253927 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.253893    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-workload-certs" pod "5aeb303b-220a-4b83-acb6-6c1004a977d1" (UID: "5aeb303b-220a-4b83-acb6-6c1004a977d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.254123 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.254099    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-workload-certs" pod "68f462dc-feb6-44af-a8af-bc45ce2e0e77" (UID: "68f462dc-feb6-44af-a8af-bc45ce2e0e77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.254396 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.254373    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istiod-ca-cert" pod "68f462dc-feb6-44af-a8af-bc45ce2e0e77" (UID: "68f462dc-feb6-44af-a8af-bc45ce2e0e77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:26.255251 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.254666    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-token" pod "a993fc51-49ae-4c3d-8c26-1f256e5064ed" (UID: "a993fc51-49ae-4c3d-8c26-1f256e5064ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:26.255251 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.254669    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-data" pod "36638d21-d7dd-40ec-9a33-83dcbbe2d7ae" (UID: "36638d21-d7dd-40ec-9a33-83dcbbe2d7ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.255251 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.254668    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-podinfo" pod "5aeb303b-220a-4b83-acb6-6c1004a977d1" (UID: "5aeb303b-220a-4b83-acb6-6c1004a977d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:23:26.255251 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.254996    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-envoy" pod "36638d21-d7dd-40ec-9a33-83dcbbe2d7ae" (UID: "36638d21-d7dd-40ec-9a33-83dcbbe2d7ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.255251 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.255022    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aeb303b-220a-4b83-acb6-6c1004a977d1-istiod-ca-cert" pod "5aeb303b-220a-4b83-acb6-6c1004a977d1" (UID: "5aeb303b-220a-4b83-acb6-6c1004a977d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:26.255645 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.255307    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-podinfo" pod "a993fc51-49ae-4c3d-8c26-1f256e5064ed" (UID: "a993fc51-49ae-4c3d-8c26-1f256e5064ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:23:26.255779 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.255698    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-data" pod "68f462dc-feb6-44af-a8af-bc45ce2e0e77" (UID: "68f462dc-feb6-44af-a8af-bc45ce2e0e77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.255972 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.255947    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-workload-certs" pod "a993fc51-49ae-4c3d-8c26-1f256e5064ed" (UID: "a993fc51-49ae-4c3d-8c26-1f256e5064ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.256249 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.256226    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istiod-ca-cert" pod "36638d21-d7dd-40ec-9a33-83dcbbe2d7ae" (UID: "36638d21-d7dd-40ec-9a33-83dcbbe2d7ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:26.256656 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.256480    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-workload-socket" pod "36638d21-d7dd-40ec-9a33-83dcbbe2d7ae" (UID: "36638d21-d7dd-40ec-9a33-83dcbbe2d7ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.256831 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.256802    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-data" pod "a993fc51-49ae-4c3d-8c26-1f256e5064ed" (UID: "a993fc51-49ae-4c3d-8c26-1f256e5064ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.257285 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.257044    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-credential-socket" pod "a993fc51-49ae-4c3d-8c26-1f256e5064ed" (UID: "a993fc51-49ae-4c3d-8c26-1f256e5064ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.257444 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.257415    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-token" pod "36638d21-d7dd-40ec-9a33-83dcbbe2d7ae" (UID: "36638d21-d7dd-40ec-9a33-83dcbbe2d7ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:26.257757 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.257714    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-envoy" pod "a993fc51-49ae-4c3d-8c26-1f256e5064ed" (UID: "a993fc51-49ae-4c3d-8c26-1f256e5064ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.257757 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.257715    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-envoy" pod "5aeb303b-220a-4b83-acb6-6c1004a977d1" (UID: "5aeb303b-220a-4b83-acb6-6c1004a977d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:26.257909 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.257785    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-kube-api-access-lmr6l" pod "36638d21-d7dd-40ec-9a33-83dcbbe2d7ae" (UID: "36638d21-d7dd-40ec-9a33-83dcbbe2d7ae"). InnerVolumeSpecName "kube-api-access-lmr6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:26.257909 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.257855    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-token" pod "5aeb303b-220a-4b83-acb6-6c1004a977d1" (UID: "5aeb303b-220a-4b83-acb6-6c1004a977d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:26.258061 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.258040    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-token" pod "68f462dc-feb6-44af-a8af-bc45ce2e0e77" (UID: "68f462dc-feb6-44af-a8af-bc45ce2e0e77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:26.259003 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.258978    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aeb303b-220a-4b83-acb6-6c1004a977d1-kube-api-access-nljkw" pod "5aeb303b-220a-4b83-acb6-6c1004a977d1" (UID: "5aeb303b-220a-4b83-acb6-6c1004a977d1"). InnerVolumeSpecName "kube-api-access-nljkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:26.259119 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.259104    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-podinfo" pod "36638d21-d7dd-40ec-9a33-83dcbbe2d7ae" (UID: "36638d21-d7dd-40ec-9a33-83dcbbe2d7ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:23:26.259650 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.259627    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-podinfo" pod "68f462dc-feb6-44af-a8af-bc45ce2e0e77" (UID: "68f462dc-feb6-44af-a8af-bc45ce2e0e77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:23:26.353641 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353613    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353641 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353639    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353649    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353659    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p24b\" (UniqueName: \"kubernetes.io/projected/68f462dc-feb6-44af-a8af-bc45ce2e0e77-kube-api-access-9p24b\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353670    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6jrh\" (UniqueName: \"kubernetes.io/projected/a993fc51-49ae-4c3d-8c26-1f256e5064ed-kube-api-access-v6jrh\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353678    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353687    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353695    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353704    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353712    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353720    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353728    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353737    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353746    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353755    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353763    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353771    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353788    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353797    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353805    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353813    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353821    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a993fc51-49ae-4c3d-8c26-1f256e5064ed-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353838    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353847    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aeb303b-220a-4b83-acb6-6c1004a977d1-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353860    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a993fc51-49ae-4c3d-8c26-1f256e5064ed-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353872    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nljkw\" (UniqueName: \"kubernetes.io/projected/5aeb303b-220a-4b83-acb6-6c1004a977d1-kube-api-access-nljkw\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353886    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmr6l\" (UniqueName: \"kubernetes.io/projected/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae-kube-api-access-lmr6l\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353905    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f462dc-feb6-44af-a8af-bc45ce2e0e77-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.353911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.353918    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aeb303b-220a-4b83-acb6-6c1004a977d1-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:26.470491 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.470423    5958 generic.go:356] "Generic (PLEG): container finished" podID="5aeb303b-220a-4b83-acb6-6c1004a977d1" containerID="c7c9950a63166d7b3f63c8eda06afef9617a8361f10e0975bef3bba17089127f" exitCode=0
Oct 30 20:23:26.470924 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.470505    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh" event={"ID":"5aeb303b-220a-4b83-acb6-6c1004a977d1","Type":"ContainerDied","Data":"c7c9950a63166d7b3f63c8eda06afef9617a8361f10e0975bef3bba17089127f"}
Oct 30 20:23:26.470924 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.470587    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh" event={"ID":"5aeb303b-220a-4b83-acb6-6c1004a977d1","Type":"ContainerDied","Data":"f1b10a73fba3e18c48e8d5b2da7fe08c7a827f5ecfbe52e62f733cbbbf0d8502"}
Oct 30 20:23:26.470924 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.470522    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"
Oct 30 20:23:26.470924 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.470617    5958 scope.go:122] "RemoveContainer" containerID="c7c9950a63166d7b3f63c8eda06afef9617a8361f10e0975bef3bba17089127f"
Oct 30 20:23:26.472483 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.472429    5958 generic.go:356] "Generic (PLEG): container finished" podID="a993fc51-49ae-4c3d-8c26-1f256e5064ed" containerID="2cdeb0eda1225c96d3d10d62205122c338b46fefce991375cd1310195fba8403" exitCode=0
Oct 30 20:23:26.472605 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.472481    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m" event={"ID":"a993fc51-49ae-4c3d-8c26-1f256e5064ed","Type":"ContainerDied","Data":"2cdeb0eda1225c96d3d10d62205122c338b46fefce991375cd1310195fba8403"}
Oct 30 20:23:26.472605 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.472545    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m" event={"ID":"a993fc51-49ae-4c3d-8c26-1f256e5064ed","Type":"ContainerDied","Data":"0bfed1831eaa8b3c4969774eb9c9f22f2e63791823dc31b5a3baf0a0976c77f9"}
Oct 30 20:23:26.472774 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.472760    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"
Oct 30 20:23:26.474867 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.474841    5958 generic.go:356] "Generic (PLEG): container finished" podID="68f462dc-feb6-44af-a8af-bc45ce2e0e77" containerID="4f904325479799b0c3a1d8f710a7c52f4d4aeab32fda4ddf3467a42a622eaf6f" exitCode=0
Oct 30 20:23:26.474995 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.474974    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9" event={"ID":"68f462dc-feb6-44af-a8af-bc45ce2e0e77","Type":"ContainerDied","Data":"4f904325479799b0c3a1d8f710a7c52f4d4aeab32fda4ddf3467a42a622eaf6f"}
Oct 30 20:23:26.475068 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.475017    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9" event={"ID":"68f462dc-feb6-44af-a8af-bc45ce2e0e77","Type":"ContainerDied","Data":"6e4b0ca3a552cfecb05ee5494a09297658740ac46165aa5550d301d1920f3227"}
Oct 30 20:23:26.475124 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.475101    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"
Oct 30 20:23:26.479715 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.479680    5958 generic.go:356] "Generic (PLEG): container finished" podID="36638d21-d7dd-40ec-9a33-83dcbbe2d7ae" containerID="d67212398c218a75ab46f08d283ed2b36dc8ca00a5d93f34cbd805cc14d10c16" exitCode=0
Oct 30 20:23:26.479715 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.479709    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll" event={"ID":"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae","Type":"ContainerDied","Data":"d67212398c218a75ab46f08d283ed2b36dc8ca00a5d93f34cbd805cc14d10c16"}
Oct 30 20:23:26.479923 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.479733    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"
Oct 30 20:23:26.479923 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.479744    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll" event={"ID":"36638d21-d7dd-40ec-9a33-83dcbbe2d7ae","Type":"ContainerDied","Data":"aada4b642a52c0e9226f0f761139459a00c78d179d81a064a7f1de420643d854"}
Oct 30 20:23:26.486278 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.486254    5958 scope.go:122] "RemoveContainer" containerID="addbc3e79ea43fcd344f36a942aa0e9cbb1b3582fb831e936ea79b331d2eece9"
Oct 30 20:23:26.501602 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.501582    5958 scope.go:122] "RemoveContainer" containerID="c7c9950a63166d7b3f63c8eda06afef9617a8361f10e0975bef3bba17089127f"
Oct 30 20:23:26.501985 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:23:26.501960    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7c9950a63166d7b3f63c8eda06afef9617a8361f10e0975bef3bba17089127f\": container with ID starting with c7c9950a63166d7b3f63c8eda06afef9617a8361f10e0975bef3bba17089127f not found: ID does not exist" containerID="c7c9950a63166d7b3f63c8eda06afef9617a8361f10e0975bef3bba17089127f"
Oct 30 20:23:26.502097 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.501991    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c9950a63166d7b3f63c8eda06afef9617a8361f10e0975bef3bba17089127f"} err="failed to get container status \"c7c9950a63166d7b3f63c8eda06afef9617a8361f10e0975bef3bba17089127f\": rpc error: code = NotFound desc = could not find container \"c7c9950a63166d7b3f63c8eda06afef9617a8361f10e0975bef3bba17089127f\": container with ID starting with c7c9950a63166d7b3f63c8eda06afef9617a8361f10e0975bef3bba17089127f not found: ID does not exist"
Oct 30 20:23:26.502097 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.502008    5958 scope.go:122] "RemoveContainer" containerID="addbc3e79ea43fcd344f36a942aa0e9cbb1b3582fb831e936ea79b331d2eece9"
Oct 30 20:23:26.502297 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:23:26.502272    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"addbc3e79ea43fcd344f36a942aa0e9cbb1b3582fb831e936ea79b331d2eece9\": container with ID starting with addbc3e79ea43fcd344f36a942aa0e9cbb1b3582fb831e936ea79b331d2eece9 not found: ID does not exist" containerID="addbc3e79ea43fcd344f36a942aa0e9cbb1b3582fb831e936ea79b331d2eece9"
Oct 30 20:23:26.502372 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.502305    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"addbc3e79ea43fcd344f36a942aa0e9cbb1b3582fb831e936ea79b331d2eece9"} err="failed to get container status \"addbc3e79ea43fcd344f36a942aa0e9cbb1b3582fb831e936ea79b331d2eece9\": rpc error: code = NotFound desc = could not find container \"addbc3e79ea43fcd344f36a942aa0e9cbb1b3582fb831e936ea79b331d2eece9\": container with ID starting with addbc3e79ea43fcd344f36a942aa0e9cbb1b3582fb831e936ea79b331d2eece9 not found: ID does not exist"
Oct 30 20:23:26.502372 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.502327    5958 scope.go:122] "RemoveContainer" containerID="2cdeb0eda1225c96d3d10d62205122c338b46fefce991375cd1310195fba8403"
Oct 30 20:23:26.511935 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.511917    5958 scope.go:122] "RemoveContainer" containerID="2cdeb0eda1225c96d3d10d62205122c338b46fefce991375cd1310195fba8403"
Oct 30 20:23:26.512202 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:23:26.512182    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cdeb0eda1225c96d3d10d62205122c338b46fefce991375cd1310195fba8403\": container with ID starting with 2cdeb0eda1225c96d3d10d62205122c338b46fefce991375cd1310195fba8403 not found: ID does not exist" containerID="2cdeb0eda1225c96d3d10d62205122c338b46fefce991375cd1310195fba8403"
Oct 30 20:23:26.512245 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.512209    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cdeb0eda1225c96d3d10d62205122c338b46fefce991375cd1310195fba8403"} err="failed to get container status \"2cdeb0eda1225c96d3d10d62205122c338b46fefce991375cd1310195fba8403\": rpc error: code = NotFound desc = could not find container \"2cdeb0eda1225c96d3d10d62205122c338b46fefce991375cd1310195fba8403\": container with ID starting with 2cdeb0eda1225c96d3d10d62205122c338b46fefce991375cd1310195fba8403 not found: ID does not exist"
Oct 30 20:23:26.512245 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.512225    5958 scope.go:122] "RemoveContainer" containerID="4f904325479799b0c3a1d8f710a7c52f4d4aeab32fda4ddf3467a42a622eaf6f"
Oct 30 20:23:26.520685 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.520667    5958 scope.go:122] "RemoveContainer" containerID="4f904325479799b0c3a1d8f710a7c52f4d4aeab32fda4ddf3467a42a622eaf6f"
Oct 30 20:23:26.520948 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:23:26.520929    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f904325479799b0c3a1d8f710a7c52f4d4aeab32fda4ddf3467a42a622eaf6f\": container with ID starting with 4f904325479799b0c3a1d8f710a7c52f4d4aeab32fda4ddf3467a42a622eaf6f not found: ID does not exist" containerID="4f904325479799b0c3a1d8f710a7c52f4d4aeab32fda4ddf3467a42a622eaf6f"
Oct 30 20:23:26.521016 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.520957    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f904325479799b0c3a1d8f710a7c52f4d4aeab32fda4ddf3467a42a622eaf6f"} err="failed to get container status \"4f904325479799b0c3a1d8f710a7c52f4d4aeab32fda4ddf3467a42a622eaf6f\": rpc error: code = NotFound desc = could not find container \"4f904325479799b0c3a1d8f710a7c52f4d4aeab32fda4ddf3467a42a622eaf6f\": container with ID starting with 4f904325479799b0c3a1d8f710a7c52f4d4aeab32fda4ddf3467a42a622eaf6f not found: ID does not exist"
Oct 30 20:23:26.521016 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.520980    5958 scope.go:122] "RemoveContainer" containerID="d67212398c218a75ab46f08d283ed2b36dc8ca00a5d93f34cbd805cc14d10c16"
Oct 30 20:23:26.529164 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.529145    5958 scope.go:122] "RemoveContainer" containerID="d67212398c218a75ab46f08d283ed2b36dc8ca00a5d93f34cbd805cc14d10c16"
Oct 30 20:23:26.529414 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:23:26.529393    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67212398c218a75ab46f08d283ed2b36dc8ca00a5d93f34cbd805cc14d10c16\": container with ID starting with d67212398c218a75ab46f08d283ed2b36dc8ca00a5d93f34cbd805cc14d10c16 not found: ID does not exist" containerID="d67212398c218a75ab46f08d283ed2b36dc8ca00a5d93f34cbd805cc14d10c16"
Oct 30 20:23:26.529519 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.529422    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67212398c218a75ab46f08d283ed2b36dc8ca00a5d93f34cbd805cc14d10c16"} err="failed to get container status \"d67212398c218a75ab46f08d283ed2b36dc8ca00a5d93f34cbd805cc14d10c16\": rpc error: code = NotFound desc = could not find container \"d67212398c218a75ab46f08d283ed2b36dc8ca00a5d93f34cbd805cc14d10c16\": container with ID starting with d67212398c218a75ab46f08d283ed2b36dc8ca00a5d93f34cbd805cc14d10c16 not found: ID does not exist"
Oct 30 20:23:26.773577 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.773540    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"]
Oct 30 20:23:26.780474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.780429    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-nspll"]
Oct 30 20:23:26.891094 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.891013    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"]
Oct 30 20:23:26.893444 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:26.893418    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-rc42m"]
Oct 30 20:23:27.006243 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:27.006197    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"]
Oct 30 20:23:27.012190 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:27.012162    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-6dwhh"]
Oct 30 20:23:27.124474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:27.123671    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"]
Oct 30 20:23:27.126523 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:27.126494    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-37732de7-openshift-default-878ccfcb5-d9sj9"]
Oct 30 20:23:27.355819 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:27.355788    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:23:27.368251 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:27.368224    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:23:27.400409 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:27.400375    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36638d21-d7dd-40ec-9a33-83dcbbe2d7ae" path="/var/lib/kubelet/pods/36638d21-d7dd-40ec-9a33-83dcbbe2d7ae/volumes"
Oct 30 20:23:27.400878 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:27.400864    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aeb303b-220a-4b83-acb6-6c1004a977d1" path="/var/lib/kubelet/pods/5aeb303b-220a-4b83-acb6-6c1004a977d1/volumes"
Oct 30 20:23:27.401293 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:27.401279    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68f462dc-feb6-44af-a8af-bc45ce2e0e77" path="/var/lib/kubelet/pods/68f462dc-feb6-44af-a8af-bc45ce2e0e77/volumes"
Oct 30 20:23:27.402044 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:27.402029    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a993fc51-49ae-4c3d-8c26-1f256e5064ed" path="/var/lib/kubelet/pods/a993fc51-49ae-4c3d-8c26-1f256e5064ed/volumes"
Oct 30 20:23:27.501495 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:27.501401    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:23:34.590520 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.590481    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main" containerID="cri-o://400a1dc92c4df55ff03dbad03fa4441d5aaa21a8842cf7fcb384dea917599a99" gracePeriod=45
Oct 30 20:23:34.639593 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.639548    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" containerID="cri-o://1cd8d31e8b7c1ce7d603a88ec2f9ac2bb420c8e93beb825787f17adb98ba0074" gracePeriod=45
Oct 30 20:23:34.759549 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.759518    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": dial tcp 10.134.0.60:8000: connect: connection refused"
Oct 30 20:23:34.916607 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.916584    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:23:34.930731 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.930710    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:23:34.958047 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.958025    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-dshm\") pod \"7804ed32-f5d5-41b7-856f-ceebcad9a306\" (UID: \"7804ed32-f5d5-41b7-856f-ceebcad9a306\") "
Oct 30 20:23:34.958139 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.958080    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-model-cache\") pod \"7804ed32-f5d5-41b7-856f-ceebcad9a306\" (UID: \"7804ed32-f5d5-41b7-856f-ceebcad9a306\") "
Oct 30 20:23:34.958139 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.958106    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68b0c566-35a5-41b8-b853-0524039ce73a-tls-certs\" (UniqueName: \"kubernetes.io/secret/68b0c566-35a5-41b8-b853-0524039ce73a-tls-certs\") pod \"68b0c566-35a5-41b8-b853-0524039ce73a\" (UID: \"68b0c566-35a5-41b8-b853-0524039ce73a\") "
Oct 30 20:23:34.958139 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.958131    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-model-cache\") pod \"68b0c566-35a5-41b8-b853-0524039ce73a\" (UID: \"68b0c566-35a5-41b8-b853-0524039ce73a\") "
Oct 30 20:23:34.958269 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.958167    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b0c566-35a5-41b8-b853-0524039ce73a-kube-api-access-8lsp2\" (UniqueName: \"kubernetes.io/projected/68b0c566-35a5-41b8-b853-0524039ce73a-kube-api-access-8lsp2\") pod \"68b0c566-35a5-41b8-b853-0524039ce73a\" (UID: \"68b0c566-35a5-41b8-b853-0524039ce73a\") "
Oct 30 20:23:34.958269 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.958204    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-tmp-dir\") pod \"7804ed32-f5d5-41b7-856f-ceebcad9a306\" (UID: \"7804ed32-f5d5-41b7-856f-ceebcad9a306\") "
Oct 30 20:23:34.958269 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.958243    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7804ed32-f5d5-41b7-856f-ceebcad9a306-tls-certs\" (UniqueName: \"kubernetes.io/secret/7804ed32-f5d5-41b7-856f-ceebcad9a306-tls-certs\") pod \"7804ed32-f5d5-41b7-856f-ceebcad9a306\" (UID: \"7804ed32-f5d5-41b7-856f-ceebcad9a306\") "
Oct 30 20:23:34.958417 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.958299    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-dshm\") pod \"68b0c566-35a5-41b8-b853-0524039ce73a\" (UID: \"68b0c566-35a5-41b8-b853-0524039ce73a\") "
Oct 30 20:23:34.958417 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.958351    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7804ed32-f5d5-41b7-856f-ceebcad9a306-kube-api-access-4k5bg\" (UniqueName: \"kubernetes.io/projected/7804ed32-f5d5-41b7-856f-ceebcad9a306-kube-api-access-4k5bg\") pod \"7804ed32-f5d5-41b7-856f-ceebcad9a306\" (UID: \"7804ed32-f5d5-41b7-856f-ceebcad9a306\") "
Oct 30 20:23:34.958417 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.958385    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-tmp-dir\") pod \"68b0c566-35a5-41b8-b853-0524039ce73a\" (UID: \"68b0c566-35a5-41b8-b853-0524039ce73a\") "
Oct 30 20:23:34.959311 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.958416    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-model-cache" pod "7804ed32-f5d5-41b7-856f-ceebcad9a306" (UID: "7804ed32-f5d5-41b7-856f-ceebcad9a306"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:34.959311 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.958490    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-model-cache" pod "68b0c566-35a5-41b8-b853-0524039ce73a" (UID: "68b0c566-35a5-41b8-b853-0524039ce73a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:34.959311 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.958517    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-home\" (UniqueName: \"kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-home\") pod \"68b0c566-35a5-41b8-b853-0524039ce73a\" (UID: \"68b0c566-35a5-41b8-b853-0524039ce73a\") "
Oct 30 20:23:34.959311 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.958517    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-tmp-dir" pod "7804ed32-f5d5-41b7-856f-ceebcad9a306" (UID: "7804ed32-f5d5-41b7-856f-ceebcad9a306"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:34.959311 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.958552    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-home\" (UniqueName: \"kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-home\") pod \"7804ed32-f5d5-41b7-856f-ceebcad9a306\" (UID: \"7804ed32-f5d5-41b7-856f-ceebcad9a306\") "
Oct 30 20:23:34.959311 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.958724    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-tmp-dir" pod "68b0c566-35a5-41b8-b853-0524039ce73a" (UID: "68b0c566-35a5-41b8-b853-0524039ce73a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:34.959311 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.958896    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-home" pod "7804ed32-f5d5-41b7-856f-ceebcad9a306" (UID: "7804ed32-f5d5-41b7-856f-ceebcad9a306"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:34.959311 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.959065    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-home" pod "68b0c566-35a5-41b8-b853-0524039ce73a" (UID: "68b0c566-35a5-41b8-b853-0524039ce73a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:34.959311 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.959080    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-home\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:34.959311 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.959097    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-model-cache\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:34.959311 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.959112    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-model-cache\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:34.959311 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.959125    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-tmp-dir\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:34.959311 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.959138    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-tmp-dir\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:34.961027 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.960998    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-dshm" pod "68b0c566-35a5-41b8-b853-0524039ce73a" (UID: "68b0c566-35a5-41b8-b853-0524039ce73a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:34.961119 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.961076    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b0c566-35a5-41b8-b853-0524039ce73a-kube-api-access-8lsp2" pod "68b0c566-35a5-41b8-b853-0524039ce73a" (UID: "68b0c566-35a5-41b8-b853-0524039ce73a"). InnerVolumeSpecName "kube-api-access-8lsp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:34.961215 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.961197    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7804ed32-f5d5-41b7-856f-ceebcad9a306-kube-api-access-4k5bg" pod "7804ed32-f5d5-41b7-856f-ceebcad9a306" (UID: "7804ed32-f5d5-41b7-856f-ceebcad9a306"). InnerVolumeSpecName "kube-api-access-4k5bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:34.961215 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.961202    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7804ed32-f5d5-41b7-856f-ceebcad9a306-tls-certs" pod "7804ed32-f5d5-41b7-856f-ceebcad9a306" (UID: "7804ed32-f5d5-41b7-856f-ceebcad9a306"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:34.961291 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.961211    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-dshm" pod "7804ed32-f5d5-41b7-856f-ceebcad9a306" (UID: "7804ed32-f5d5-41b7-856f-ceebcad9a306"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:34.962466 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:34.962432    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68b0c566-35a5-41b8-b853-0524039ce73a-tls-certs" pod "68b0c566-35a5-41b8-b853-0524039ce73a" (UID: "68b0c566-35a5-41b8-b853-0524039ce73a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:35.060606 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:35.060566    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lsp2\" (UniqueName: \"kubernetes.io/projected/68b0c566-35a5-41b8-b853-0524039ce73a-kube-api-access-8lsp2\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:35.060606 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:35.060607    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7804ed32-f5d5-41b7-856f-ceebcad9a306-tls-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:35.060848 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:35.060623    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-dshm\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:35.060848 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:35.060639    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k5bg\" (UniqueName: \"kubernetes.io/projected/7804ed32-f5d5-41b7-856f-ceebcad9a306-kube-api-access-4k5bg\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:35.060848 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:35.060655    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68b0c566-35a5-41b8-b853-0524039ce73a-home\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:35.060848 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:35.060668    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7804ed32-f5d5-41b7-856f-ceebcad9a306-dshm\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:35.060848 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:35.060681    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b0c566-35a5-41b8-b853-0524039ce73a-tls-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:35.544964 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:35.544929    5958 generic.go:356] "Generic (PLEG): container finished" podID="68b0c566-35a5-41b8-b853-0524039ce73a" containerID="400a1dc92c4df55ff03dbad03fa4441d5aaa21a8842cf7fcb384dea917599a99" exitCode=0
Oct 30 20:23:35.545093 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:35.544972    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"
Oct 30 20:23:35.545093 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:35.544977    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" event={"ID":"68b0c566-35a5-41b8-b853-0524039ce73a","Type":"ContainerDied","Data":"400a1dc92c4df55ff03dbad03fa4441d5aaa21a8842cf7fcb384dea917599a99"}
Oct 30 20:23:35.545093 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:35.545023    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25" event={"ID":"68b0c566-35a5-41b8-b853-0524039ce73a","Type":"ContainerDied","Data":"67450dccc31e7d6ad659c1a66998a03adba1cc237d579891c13dbba946d9dae3"}
Oct 30 20:23:35.545093 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:35.545046    5958 scope.go:122] "RemoveContainer" containerID="400a1dc92c4df55ff03dbad03fa4441d5aaa21a8842cf7fcb384dea917599a99"
Oct 30 20:23:35.546738 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:35.546677    5958 generic.go:356] "Generic (PLEG): container finished" podID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerID="1cd8d31e8b7c1ce7d603a88ec2f9ac2bb420c8e93beb825787f17adb98ba0074" exitCode=0
Oct 30 20:23:35.546864 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:35.546742    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"
Oct 30 20:23:35.546864 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:35.546740    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" event={"ID":"7804ed32-f5d5-41b7-856f-ceebcad9a306","Type":"ContainerDied","Data":"1cd8d31e8b7c1ce7d603a88ec2f9ac2bb420c8e93beb825787f17adb98ba0074"}
Oct 30 20:23:35.546864 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:35.546847    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg" event={"ID":"7804ed32-f5d5-41b7-856f-ceebcad9a306","Type":"ContainerDied","Data":"ffaf9dd5c8e8e8aa8b4b47524cd926b2218f2488dfc007ed955136618558453a"}
Oct 30 20:23:35.555609 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:35.555593    5958 scope.go:122] "RemoveContainer" containerID="400a1dc92c4df55ff03dbad03fa4441d5aaa21a8842cf7fcb384dea917599a99"
Oct 30 20:23:35.555882 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:23:35.555862    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"400a1dc92c4df55ff03dbad03fa4441d5aaa21a8842cf7fcb384dea917599a99\": container with ID starting with 400a1dc92c4df55ff03dbad03fa4441d5aaa21a8842cf7fcb384dea917599a99 not found: ID does not exist" containerID="400a1dc92c4df55ff03dbad03fa4441d5aaa21a8842cf7fcb384dea917599a99"
Oct 30 20:23:35.555941 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:35.555887    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"400a1dc92c4df55ff03dbad03fa4441d5aaa21a8842cf7fcb384dea917599a99"} err="failed to get container status \"400a1dc92c4df55ff03dbad03fa4441d5aaa21a8842cf7fcb384dea917599a99\": rpc error: code = NotFound desc = could not find container \"400a1dc92c4df55ff03dbad03fa4441d5aaa21a8842cf7fcb384dea917599a99\": container with ID starting with 400a1dc92c4df55ff03dbad03fa4441d5aaa21a8842cf7fcb384dea917599a99 not found: ID does not exist"
Oct 30 20:23:35.555941 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:35.555905    5958 scope.go:122] "RemoveContainer" containerID="1cd8d31e8b7c1ce7d603a88ec2f9ac2bb420c8e93beb825787f17adb98ba0074"
Oct 30 20:23:35.564317 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:35.564294    5958 scope.go:122] "RemoveContainer" containerID="1cd8d31e8b7c1ce7d603a88ec2f9ac2bb420c8e93beb825787f17adb98ba0074"
Oct 30 20:23:35.564582 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:23:35.564563    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd8d31e8b7c1ce7d603a88ec2f9ac2bb420c8e93beb825787f17adb98ba0074\": container with ID starting with 1cd8d31e8b7c1ce7d603a88ec2f9ac2bb420c8e93beb825787f17adb98ba0074 not found: ID does not exist" containerID="1cd8d31e8b7c1ce7d603a88ec2f9ac2bb420c8e93beb825787f17adb98ba0074"
Oct 30 20:23:35.564636 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:35.564589    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd8d31e8b7c1ce7d603a88ec2f9ac2bb420c8e93beb825787f17adb98ba0074"} err="failed to get container status \"1cd8d31e8b7c1ce7d603a88ec2f9ac2bb420c8e93beb825787f17adb98ba0074\": rpc error: code = NotFound desc = could not find container \"1cd8d31e8b7c1ce7d603a88ec2f9ac2bb420c8e93beb825787f17adb98ba0074\": container with ID starting with 1cd8d31e8b7c1ce7d603a88ec2f9ac2bb420c8e93beb825787f17adb98ba0074 not found: ID does not exist"
Oct 30 20:23:35.710865 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:35.710812    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"]
Oct 30 20:23:35.760398 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:35.760364    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-bebb42/simulator-w4-bebb42-kserve-b759cb9d-h5xpg"]
Oct 30 20:23:35.826179 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:35.826149    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"]
Oct 30 20:23:35.830380 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:35.830357    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-bebb42/distinct-2-w4-bebb42-kserve-797567597-jdv25"]
Oct 30 20:23:36.606429 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:23:36.606392    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:23:37.398776 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:37.398743    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" path="/var/lib/kubelet/pods/68b0c566-35a5-41b8-b853-0524039ce73a/volumes"
Oct 30 20:23:37.399200 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:37.399186    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" path="/var/lib/kubelet/pods/7804ed32-f5d5-41b7-856f-ceebcad9a306/volumes"
Oct 30 20:23:39.947009 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.946976    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h"]
Oct 30 20:23:39.948082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.947822    5958 cpu_manager.go:414] "containerMap: removing container" podUID="36638d21-d7dd-40ec-9a33-83dcbbe2d7ae" containerName="istio-proxy"
Oct 30 20:23:39.948082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.947840    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36638d21-d7dd-40ec-9a33-83dcbbe2d7ae" containerName="istio-proxy"
Oct 30 20:23:39.948082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.947863    5958 cpu_manager.go:414] "containerMap: removing container" podUID="68f462dc-feb6-44af-a8af-bc45ce2e0e77" containerName="istio-proxy"
Oct 30 20:23:39.948082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.947871    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68f462dc-feb6-44af-a8af-bc45ce2e0e77" containerName="istio-proxy"
Oct 30 20:23:39.948082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.947892    5958 cpu_manager.go:414] "containerMap: removing container" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main"
Oct 30 20:23:39.948082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.947898    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main"
Oct 30 20:23:39.948082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.947910    5958 cpu_manager.go:414] "containerMap: removing container" podUID="5aeb303b-220a-4b83-acb6-6c1004a977d1" containerName="istio-proxy"
Oct 30 20:23:39.948082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.947916    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aeb303b-220a-4b83-acb6-6c1004a977d1" containerName="istio-proxy"
Oct 30 20:23:39.948082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.947937    5958 cpu_manager.go:414] "containerMap: removing container" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main"
Oct 30 20:23:39.948082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.947944    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main"
Oct 30 20:23:39.948082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.947962    5958 cpu_manager.go:414] "containerMap: removing container" podUID="56ec387e-8e34-472e-b8f5-032a57c517e9" containerName="istio-proxy"
Oct 30 20:23:39.948082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.947968    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56ec387e-8e34-472e-b8f5-032a57c517e9" containerName="istio-proxy"
Oct 30 20:23:39.948082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.947977    5958 cpu_manager.go:414] "containerMap: removing container" podUID="5aeb303b-220a-4b83-acb6-6c1004a977d1" containerName="istio-proxy"
Oct 30 20:23:39.948082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.947983    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aeb303b-220a-4b83-acb6-6c1004a977d1" containerName="istio-proxy"
Oct 30 20:23:39.948082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.947994    5958 cpu_manager.go:414] "containerMap: removing container" podUID="a993fc51-49ae-4c3d-8c26-1f256e5064ed" containerName="istio-proxy"
Oct 30 20:23:39.948082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.948001    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a993fc51-49ae-4c3d-8c26-1f256e5064ed" containerName="istio-proxy"
Oct 30 20:23:39.948082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.948036    5958 cpu_manager.go:414] "containerMap: removing container" podUID="56ec387e-8e34-472e-b8f5-032a57c517e9" containerName="istio-proxy"
Oct 30 20:23:39.948082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.948042    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56ec387e-8e34-472e-b8f5-032a57c517e9" containerName="istio-proxy"
Oct 30 20:23:39.948934 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.948166    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="56ec387e-8e34-472e-b8f5-032a57c517e9" containerName="istio-proxy"
Oct 30 20:23:39.948934 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.948176    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aeb303b-220a-4b83-acb6-6c1004a977d1" containerName="istio-proxy"
Oct 30 20:23:39.948934 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.948188    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="a993fc51-49ae-4c3d-8c26-1f256e5064ed" containerName="istio-proxy"
Oct 30 20:23:39.948934 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.948200    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="56ec387e-8e34-472e-b8f5-032a57c517e9" containerName="istio-proxy"
Oct 30 20:23:39.948934 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.948211    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="36638d21-d7dd-40ec-9a33-83dcbbe2d7ae" containerName="istio-proxy"
Oct 30 20:23:39.948934 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.948223    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="68f462dc-feb6-44af-a8af-bc45ce2e0e77" containerName="istio-proxy"
Oct 30 20:23:39.948934 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.948233    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="7804ed32-f5d5-41b7-856f-ceebcad9a306" containerName="main"
Oct 30 20:23:39.948934 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.948241    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b0c566-35a5-41b8-b853-0524039ce73a" containerName="main"
Oct 30 20:23:39.968015 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:39.967993    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h"
Oct 30 20:23:40.011236 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:40.011210    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h"]
Oct 30 20:23:40.017173 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:40.017148    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e12ca895-aa85-41db-89bf-d9912ba799f9-metrics-certs\") pod \"maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h\" (UID: \"e12ca895-aa85-41db-89bf-d9912ba799f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h"
Oct 30 20:23:40.017299 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:40.017227    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmskn\" (UniqueName: \"kubernetes.io/projected/e12ca895-aa85-41db-89bf-d9912ba799f9-kube-api-access-fmskn\") pod \"maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h\" (UID: \"e12ca895-aa85-41db-89bf-d9912ba799f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h"
Oct 30 20:23:40.017299 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:40.017281    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12ca895-aa85-41db-89bf-d9912ba799f9-maas-api-tls\") pod \"maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h\" (UID: \"e12ca895-aa85-41db-89bf-d9912ba799f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h"
Oct 30 20:23:40.017430 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:40.017401    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12ca895-aa85-41db-89bf-d9912ba799f9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h\" (UID: \"e12ca895-aa85-41db-89bf-d9912ba799f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h"
Oct 30 20:23:40.020998 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:40.020973    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-453277-serving-cert\""
Oct 30 20:23:40.118507 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:40.118480    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmskn\" (UniqueName: \"kubernetes.io/projected/e12ca895-aa85-41db-89bf-d9912ba799f9-kube-api-access-fmskn\") pod \"maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h\" (UID: \"e12ca895-aa85-41db-89bf-d9912ba799f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h"
Oct 30 20:23:40.118664 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:40.118564    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12ca895-aa85-41db-89bf-d9912ba799f9-maas-api-tls\") pod \"maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h\" (UID: \"e12ca895-aa85-41db-89bf-d9912ba799f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h"
Oct 30 20:23:40.118737 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:40.118686    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12ca895-aa85-41db-89bf-d9912ba799f9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h\" (UID: \"e12ca895-aa85-41db-89bf-d9912ba799f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h"
Oct 30 20:23:40.118798 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:40.118763    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e12ca895-aa85-41db-89bf-d9912ba799f9-metrics-certs\") pod \"maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h\" (UID: \"e12ca895-aa85-41db-89bf-d9912ba799f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h"
Oct 30 20:23:40.118798 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:23:40.118789    5958 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-serving-cert: secret "maas-api-e2e-inf-b-453277-serving-cert" not found
Oct 30 20:23:40.118904 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:23:40.118892    5958 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12ca895-aa85-41db-89bf-d9912ba799f9-maas-api-tls podName:e12ca895-aa85-41db-89bf-d9912ba799f9 nodeName:}" failed. No retries permitted until 2026-10-30 20:23:40.618873339 +0000 UTC m=+2459.351088721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e12ca895-aa85-41db-89bf-d9912ba799f9-maas-api-tls") pod "maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h" (UID: "e12ca895-aa85-41db-89bf-d9912ba799f9") : secret "maas-api-e2e-inf-b-453277-serving-cert" not found
Oct 30 20:23:40.119492 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:40.119443    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12ca895-aa85-41db-89bf-d9912ba799f9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h\" (UID: \"e12ca895-aa85-41db-89bf-d9912ba799f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h"
Oct 30 20:23:40.121508 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:40.121443    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e12ca895-aa85-41db-89bf-d9912ba799f9-metrics-certs\") pod \"maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h\" (UID: \"e12ca895-aa85-41db-89bf-d9912ba799f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h"
Oct 30 20:23:40.179484 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:40.179440    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmskn\" (UniqueName: \"kubernetes.io/projected/e12ca895-aa85-41db-89bf-d9912ba799f9-kube-api-access-fmskn\") pod \"maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h\" (UID: \"e12ca895-aa85-41db-89bf-d9912ba799f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h"
Oct 30 20:23:40.625399 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:40.625363    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12ca895-aa85-41db-89bf-d9912ba799f9-maas-api-tls\") pod \"maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h\" (UID: \"e12ca895-aa85-41db-89bf-d9912ba799f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h"
Oct 30 20:23:40.628281 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:40.628256    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12ca895-aa85-41db-89bf-d9912ba799f9-maas-api-tls\") pod \"maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h\" (UID: \"e12ca895-aa85-41db-89bf-d9912ba799f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h"
Oct 30 20:23:40.881428 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:40.881354    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h"
Oct 30 20:23:41.276209 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:41.276185    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h"]
Oct 30 20:23:41.278427 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:23:41.278399    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12ca895_aa85_41db_89bf_d9912ba799f9.slice/crio-204df3d867834dccefb1bd104c7a03e6771a53890bb5b6e0ec7c0157838cf367 WatchSource:0}: Error finding container 204df3d867834dccefb1bd104c7a03e6771a53890bb5b6e0ec7c0157838cf367: Status 404 returned error can't find the container with id 204df3d867834dccefb1bd104c7a03e6771a53890bb5b6e0ec7c0157838cf367
Oct 30 20:23:41.280158 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:41.280139    5958 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:23:41.589280 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:41.589250    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h" event={"ID":"e12ca895-aa85-41db-89bf-d9912ba799f9","Type":"ContainerStarted","Data":"204df3d867834dccefb1bd104c7a03e6771a53890bb5b6e0ec7c0157838cf367"}
Oct 30 20:23:42.597023 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:42.596987    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h" event={"ID":"e12ca895-aa85-41db-89bf-d9912ba799f9","Type":"ContainerStarted","Data":"7bef49e364f7eb5f614947b48aa790585af70bba4029bfee7451e9f7b7494242"}
Oct 30 20:23:42.597388 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:42.597109    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h"
Oct 30 20:23:42.715102 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:42.714999    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h" podStartSLOduration=2.857859499 podStartE2EDuration="3.714982485s" totalImagesPullingTime="857.122986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:42.714366987 +0000 UTC m=+2461.446582377" watchObservedRunningTime="2026-10-30 20:23:42.714982485 +0000 UTC m=+2461.447197875"
Oct 30 20:23:43.668540 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:43.668508    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"]
Oct 30 20:23:43.718192 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:43.718158    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"]
Oct 30 20:23:48.608596 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:48.608564    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h"
Oct 30 20:23:50.756361 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:50.756326    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb"]
Oct 30 20:23:50.869470 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:50.869425    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp"]
Oct 30 20:23:55.757422 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:55.757369    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb" podUID="6bdc83b4-72fc-42b0-83f9-740a2ce1cb71" containerName="payload-processing" containerID="cri-o://c0e6d5c2c21b5c54b27450229391a7a70639939d1862f2a82cb4ac8d7bc5c733" gracePeriod=25
Oct 30 20:23:55.869798 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:55.869744    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp" podUID="fbd4cbac-918f-4517-bdb7-9c4fcecf2401" containerName="payload-pre-processing" containerID="cri-o://44bf9565b02e5439fd7ffebcbab406f1c86a902e4f13789625c21bb621d1dbdf" gracePeriod=25
Oct 30 20:23:58.690414 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:58.690360    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main" containerID="cri-o://4d4201fe8e4f622e949bd0199f54266951a843b213469b78b869852ddf1112c1" gracePeriod=45
Oct 30 20:23:58.737978 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:58.737948    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main" containerID="cri-o://7fbcc9a64dd557a0f9d630a704a13c5a9fd6e6bb44db9b2eb3bd317c7e83f347" gracePeriod=45
Oct 30 20:23:58.997704 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:58.997663    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:23:59.034896 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.034869    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-tmp-dir\") pod \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\" (UID: \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\") "
Oct 30 20:23:59.034999 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.034934    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-kube-api-access-p5tb9\" (UniqueName: \"kubernetes.io/projected/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-kube-api-access-p5tb9\") pod \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\" (UID: \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\") "
Oct 30 20:23:59.034999 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.034971    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-dshm\") pod \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\" (UID: \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\") "
Oct 30 20:23:59.035121 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.034996    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-model-cache\") pod \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\" (UID: \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\") "
Oct 30 20:23:59.035121 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.035071    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-tls-certs\") pod \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\" (UID: \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\") "
Oct 30 20:23:59.035121 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.035100    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-tmp-dir" pod "fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" (UID: "fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:59.035276 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.035132    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-home\" (UniqueName: \"kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-home\") pod \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\" (UID: \"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480\") "
Oct 30 20:23:59.035540 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.035513    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-home" pod "fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" (UID: "fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:59.035734 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.035554    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-model-cache" pod "fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" (UID: "fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:59.035885 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.035769    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-home\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:59.035885 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.035783    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-tmp-dir\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:59.035885 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.035792    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-model-cache\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:59.037509 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.037484    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-tls-certs" pod "fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" (UID: "fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:59.037707 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.037680    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-dshm" pod "fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" (UID: "fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:59.037805 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.037687    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-kube-api-access-p5tb9" pod "fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" (UID: "fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480"). InnerVolumeSpecName "kube-api-access-p5tb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:59.041829 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.041807    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 20:23:59.136759 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.136728    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-model-cache\") pod \"55e64b7b-c7ad-474e-beeb-a8b221deb328\" (UID: \"55e64b7b-c7ad-474e-beeb-a8b221deb328\") "
Oct 30 20:23:59.136923 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.136778    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e64b7b-c7ad-474e-beeb-a8b221deb328-kube-api-access-t4ppw\" (UniqueName: \"kubernetes.io/projected/55e64b7b-c7ad-474e-beeb-a8b221deb328-kube-api-access-t4ppw\") pod \"55e64b7b-c7ad-474e-beeb-a8b221deb328\" (UID: \"55e64b7b-c7ad-474e-beeb-a8b221deb328\") "
Oct 30 20:23:59.136923 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.136808    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-dshm\" (UniqueName: \"kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-dshm\") pod \"55e64b7b-c7ad-474e-beeb-a8b221deb328\" (UID: \"55e64b7b-c7ad-474e-beeb-a8b221deb328\") "
Oct 30 20:23:59.137052 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.137004    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55e64b7b-c7ad-474e-beeb-a8b221deb328-tls-certs\" (UniqueName: \"kubernetes.io/secret/55e64b7b-c7ad-474e-beeb-a8b221deb328-tls-certs\") pod \"55e64b7b-c7ad-474e-beeb-a8b221deb328\" (UID: \"55e64b7b-c7ad-474e-beeb-a8b221deb328\") "
Oct 30 20:23:59.137052 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.137033    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-model-cache" pod "55e64b7b-c7ad-474e-beeb-a8b221deb328" (UID: "55e64b7b-c7ad-474e-beeb-a8b221deb328"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:59.137174 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.137055    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-tmp-dir\") pod \"55e64b7b-c7ad-474e-beeb-a8b221deb328\" (UID: \"55e64b7b-c7ad-474e-beeb-a8b221deb328\") "
Oct 30 20:23:59.137174 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.137082    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-home\" (UniqueName: \"kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-home\") pod \"55e64b7b-c7ad-474e-beeb-a8b221deb328\" (UID: \"55e64b7b-c7ad-474e-beeb-a8b221deb328\") "
Oct 30 20:23:59.137490 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.137444    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-tmp-dir" pod "55e64b7b-c7ad-474e-beeb-a8b221deb328" (UID: "55e64b7b-c7ad-474e-beeb-a8b221deb328"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:59.137708 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.137524    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-home" pod "55e64b7b-c7ad-474e-beeb-a8b221deb328" (UID: "55e64b7b-c7ad-474e-beeb-a8b221deb328"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:59.137708 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.137674    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-tls-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:59.137870 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.137808    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-tmp-dir\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:59.137870 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.137860    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-home\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:59.137980 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.137874    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-model-cache\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:59.137980 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.137888    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5tb9\" (UniqueName: \"kubernetes.io/projected/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-kube-api-access-p5tb9\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:59.137980 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.137901    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480-dshm\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:59.139536 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.139508    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55e64b7b-c7ad-474e-beeb-a8b221deb328-tls-certs" pod "55e64b7b-c7ad-474e-beeb-a8b221deb328" (UID: "55e64b7b-c7ad-474e-beeb-a8b221deb328"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:59.140040 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.140003    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-dshm" pod "55e64b7b-c7ad-474e-beeb-a8b221deb328" (UID: "55e64b7b-c7ad-474e-beeb-a8b221deb328"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:59.140269 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.140244    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e64b7b-c7ad-474e-beeb-a8b221deb328-kube-api-access-t4ppw" pod "55e64b7b-c7ad-474e-beeb-a8b221deb328" (UID: "55e64b7b-c7ad-474e-beeb-a8b221deb328"). InnerVolumeSpecName "kube-api-access-t4ppw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:59.239240 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.239170    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e64b7b-c7ad-474e-beeb-a8b221deb328-tls-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:59.239240 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.239197    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4ppw\" (UniqueName: \"kubernetes.io/projected/55e64b7b-c7ad-474e-beeb-a8b221deb328-kube-api-access-t4ppw\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:59.239240 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.239207    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55e64b7b-c7ad-474e-beeb-a8b221deb328-dshm\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:59.716436 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.716408    5958 generic.go:356] "Generic (PLEG): container finished" podID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerID="7fbcc9a64dd557a0f9d630a704a13c5a9fd6e6bb44db9b2eb3bd317c7e83f347" exitCode=0
Oct 30 20:23:59.716823 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.716493    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" event={"ID":"55e64b7b-c7ad-474e-beeb-a8b221deb328","Type":"ContainerDied","Data":"7fbcc9a64dd557a0f9d630a704a13c5a9fd6e6bb44db9b2eb3bd317c7e83f347"}
Oct 30 20:23:59.716823 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.716512    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"
Oct 30 20:23:59.716823 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.716558    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh" event={"ID":"55e64b7b-c7ad-474e-beeb-a8b221deb328","Type":"ContainerDied","Data":"dbf08008b0914cf5f2374f69b4cb4890ea8a8538230b5abcdcfaf39651611f9e"}
Oct 30 20:23:59.716823 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.716585    5958 scope.go:122] "RemoveContainer" containerID="7fbcc9a64dd557a0f9d630a704a13c5a9fd6e6bb44db9b2eb3bd317c7e83f347"
Oct 30 20:23:59.718161 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.718080    5958 generic.go:356] "Generic (PLEG): container finished" podID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerID="4d4201fe8e4f622e949bd0199f54266951a843b213469b78b869852ddf1112c1" exitCode=0
Oct 30 20:23:59.718161 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.718115    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" event={"ID":"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480","Type":"ContainerDied","Data":"4d4201fe8e4f622e949bd0199f54266951a843b213469b78b869852ddf1112c1"}
Oct 30 20:23:59.718161 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.718131    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"
Oct 30 20:23:59.718161 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.718150    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6" event={"ID":"fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480","Type":"ContainerDied","Data":"c49c930a42bef8836f28f701f56c2769b0a4bfc876928dc5206aa86669c0c21a"}
Oct 30 20:23:59.730222 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.730201    5958 scope.go:122] "RemoveContainer" containerID="7fbcc9a64dd557a0f9d630a704a13c5a9fd6e6bb44db9b2eb3bd317c7e83f347"
Oct 30 20:23:59.730921 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:23:59.730894    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fbcc9a64dd557a0f9d630a704a13c5a9fd6e6bb44db9b2eb3bd317c7e83f347\": container with ID starting with 7fbcc9a64dd557a0f9d630a704a13c5a9fd6e6bb44db9b2eb3bd317c7e83f347 not found: ID does not exist" containerID="7fbcc9a64dd557a0f9d630a704a13c5a9fd6e6bb44db9b2eb3bd317c7e83f347"
Oct 30 20:23:59.731041 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.730930    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fbcc9a64dd557a0f9d630a704a13c5a9fd6e6bb44db9b2eb3bd317c7e83f347"} err="failed to get container status \"7fbcc9a64dd557a0f9d630a704a13c5a9fd6e6bb44db9b2eb3bd317c7e83f347\": rpc error: code = NotFound desc = could not find container \"7fbcc9a64dd557a0f9d630a704a13c5a9fd6e6bb44db9b2eb3bd317c7e83f347\": container with ID starting with 7fbcc9a64dd557a0f9d630a704a13c5a9fd6e6bb44db9b2eb3bd317c7e83f347 not found: ID does not exist"
Oct 30 20:23:59.731041 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.730952    5958 scope.go:122] "RemoveContainer" containerID="4d4201fe8e4f622e949bd0199f54266951a843b213469b78b869852ddf1112c1"
Oct 30 20:23:59.740807 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.740783    5958 scope.go:122] "RemoveContainer" containerID="4d4201fe8e4f622e949bd0199f54266951a843b213469b78b869852ddf1112c1"
Oct 30 20:23:59.741053 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:23:59.741034    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d4201fe8e4f622e949bd0199f54266951a843b213469b78b869852ddf1112c1\": container with ID starting with 4d4201fe8e4f622e949bd0199f54266951a843b213469b78b869852ddf1112c1 not found: ID does not exist" containerID="4d4201fe8e4f622e949bd0199f54266951a843b213469b78b869852ddf1112c1"
Oct 30 20:23:59.741113 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.741059    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d4201fe8e4f622e949bd0199f54266951a843b213469b78b869852ddf1112c1"} err="failed to get container status \"4d4201fe8e4f622e949bd0199f54266951a843b213469b78b869852ddf1112c1\": rpc error: code = NotFound desc = could not find container \"4d4201fe8e4f622e949bd0199f54266951a843b213469b78b869852ddf1112c1\": container with ID starting with 4d4201fe8e4f622e949bd0199f54266951a843b213469b78b869852ddf1112c1 not found: ID does not exist"
Oct 30 20:23:59.888652 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.888621    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"]
Oct 30 20:23:59.893263 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:23:59.893239    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-f75272/distinct-w6-f75272-kserve-7fb9554bc4-bvnk6"]
Oct 30 20:24:00.051511 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:00.051435    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"]
Oct 30 20:24:00.055684 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:00.055662    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-f75272/simulator-w6-f75272-kserve-685889b95d-tp9gh"]
Oct 30 20:24:01.399504 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:01.399440    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" path="/var/lib/kubelet/pods/55e64b7b-c7ad-474e-beeb-a8b221deb328/volumes"
Oct 30 20:24:01.399957 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:01.399941    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" path="/var/lib/kubelet/pods/fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480/volumes"
Oct 30 20:24:03.271012 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.270983    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"]
Oct 30 20:24:03.271626 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.271610    5958 cpu_manager.go:414] "containerMap: removing container" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main"
Oct 30 20:24:03.271685 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.271627    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main"
Oct 30 20:24:03.271685 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.271654    5958 cpu_manager.go:414] "containerMap: removing container" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main"
Oct 30 20:24:03.271685 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.271661    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main"
Oct 30 20:24:03.271782 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.271756    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb0f89dc-ae94-44d8-ac5c-f7aa9cfae480" containerName="main"
Oct 30 20:24:03.271782 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.271778    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aeb303b-220a-4b83-acb6-6c1004a977d1" containerName="istio-proxy"
Oct 30 20:24:03.271847 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.271787    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="55e64b7b-c7ad-474e-beeb-a8b221deb328" containerName="main"
Oct 30 20:24:03.276422 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.276403    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:24:03.382364 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.382328    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18ca8088-e3e8-4bd0-a302-2aab153ef504-tls-certs\") pod \"test-model-a2386c-kserve-67cd4dfc85-4v7q6\" (UID: \"18ca8088-e3e8-4bd0-a302-2aab153ef504\") " pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:24:03.382575 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.382379    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slbnd\" (UniqueName: \"kubernetes.io/projected/18ca8088-e3e8-4bd0-a302-2aab153ef504-kube-api-access-slbnd\") pod \"test-model-a2386c-kserve-67cd4dfc85-4v7q6\" (UID: \"18ca8088-e3e8-4bd0-a302-2aab153ef504\") " pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:24:03.382575 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.382551    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-tmp-dir\") pod \"test-model-a2386c-kserve-67cd4dfc85-4v7q6\" (UID: \"18ca8088-e3e8-4bd0-a302-2aab153ef504\") " pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:24:03.382575 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.382566    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a2386c\"/\"kube-root-ca.crt\""
Oct 30 20:24:03.382726 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.382564    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a2386c\"/\"openshift-service-ca.crt\""
Oct 30 20:24:03.382726 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.382602    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a2386c\"/\"default-dockercfg-pltmz\""
Oct 30 20:24:03.382726 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.382634    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a2386c\"/\"test-model-a2386c-kserve-self-signed-certs\""
Oct 30 20:24:03.382726 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.382688    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-model-cache\") pod \"test-model-a2386c-kserve-67cd4dfc85-4v7q6\" (UID: \"18ca8088-e3e8-4bd0-a302-2aab153ef504\") " pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:24:03.382726 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.382709    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-dshm\") pod \"test-model-a2386c-kserve-67cd4dfc85-4v7q6\" (UID: \"18ca8088-e3e8-4bd0-a302-2aab153ef504\") " pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:24:03.382890 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.382729    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-home\") pod \"test-model-a2386c-kserve-67cd4dfc85-4v7q6\" (UID: \"18ca8088-e3e8-4bd0-a302-2aab153ef504\") " pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:24:03.385093 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.385068    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"]
Oct 30 20:24:03.483921 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.483884    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18ca8088-e3e8-4bd0-a302-2aab153ef504-tls-certs\") pod \"test-model-a2386c-kserve-67cd4dfc85-4v7q6\" (UID: \"18ca8088-e3e8-4bd0-a302-2aab153ef504\") " pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:24:03.484100 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.483962    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slbnd\" (UniqueName: \"kubernetes.io/projected/18ca8088-e3e8-4bd0-a302-2aab153ef504-kube-api-access-slbnd\") pod \"test-model-a2386c-kserve-67cd4dfc85-4v7q6\" (UID: \"18ca8088-e3e8-4bd0-a302-2aab153ef504\") " pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:24:03.484100 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.484031    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-tmp-dir\") pod \"test-model-a2386c-kserve-67cd4dfc85-4v7q6\" (UID: \"18ca8088-e3e8-4bd0-a302-2aab153ef504\") " pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:24:03.484173 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.484161    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-model-cache\") pod \"test-model-a2386c-kserve-67cd4dfc85-4v7q6\" (UID: \"18ca8088-e3e8-4bd0-a302-2aab153ef504\") " pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:24:03.484207 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.484195    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-dshm\") pod \"test-model-a2386c-kserve-67cd4dfc85-4v7q6\" (UID: \"18ca8088-e3e8-4bd0-a302-2aab153ef504\") " pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:24:03.484244 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.484224    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-home\") pod \"test-model-a2386c-kserve-67cd4dfc85-4v7q6\" (UID: \"18ca8088-e3e8-4bd0-a302-2aab153ef504\") " pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:24:03.484499 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.484437    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-tmp-dir\") pod \"test-model-a2386c-kserve-67cd4dfc85-4v7q6\" (UID: \"18ca8088-e3e8-4bd0-a302-2aab153ef504\") " pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:24:03.484623 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.484542    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-model-cache\") pod \"test-model-a2386c-kserve-67cd4dfc85-4v7q6\" (UID: \"18ca8088-e3e8-4bd0-a302-2aab153ef504\") " pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:24:03.484679 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.484622    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-home\") pod \"test-model-a2386c-kserve-67cd4dfc85-4v7q6\" (UID: \"18ca8088-e3e8-4bd0-a302-2aab153ef504\") " pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:24:03.486539 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.486507    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-dshm\") pod \"test-model-a2386c-kserve-67cd4dfc85-4v7q6\" (UID: \"18ca8088-e3e8-4bd0-a302-2aab153ef504\") " pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:24:03.486637 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.486583    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18ca8088-e3e8-4bd0-a302-2aab153ef504-tls-certs\") pod \"test-model-a2386c-kserve-67cd4dfc85-4v7q6\" (UID: \"18ca8088-e3e8-4bd0-a302-2aab153ef504\") " pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:24:03.546247 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.546162    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slbnd\" (UniqueName: \"kubernetes.io/projected/18ca8088-e3e8-4bd0-a302-2aab153ef504-kube-api-access-slbnd\") pod \"test-model-a2386c-kserve-67cd4dfc85-4v7q6\" (UID: \"18ca8088-e3e8-4bd0-a302-2aab153ef504\") " pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:24:03.595326 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.595299    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:24:03.971866 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:03.971842    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"]
Oct 30 20:24:03.974036 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:24:03.974009    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18ca8088_e3e8_4bd0_a302_2aab153ef504.slice/crio-d0a3a8a744266463f484c0f32552861b7cccbd4b27979eed5a27aff8342cb75e WatchSource:0}: Error finding container d0a3a8a744266463f484c0f32552861b7cccbd4b27979eed5a27aff8342cb75e: Status 404 returned error can't find the container with id d0a3a8a744266463f484c0f32552861b7cccbd4b27979eed5a27aff8342cb75e
Oct 30 20:24:04.678681 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:04.678649    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4269370-x8n62"]
Oct 30 20:24:04.683315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:04.683277    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4269370-x8n62"
Oct 30 20:24:04.742291 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:04.742257    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4269370-x8n62"]
Oct 30 20:24:04.770754 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:04.770716    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6" event={"ID":"18ca8088-e3e8-4bd0-a302-2aab153ef504","Type":"ContainerStarted","Data":"a9006c45f2fb51cefff2236cdbe66c3e03b547b7f0d52626e4f203bf24fd5d46"}
Oct 30 20:24:04.770908 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:04.770770    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6" event={"ID":"18ca8088-e3e8-4bd0-a302-2aab153ef504","Type":"ContainerStarted","Data":"d0a3a8a744266463f484c0f32552861b7cccbd4b27979eed5a27aff8342cb75e"}
Oct 30 20:24:04.787861 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:04.787837    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:24:04.788071 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:04.788055    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:24:04.801685 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:04.801659    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2356c8f-e080-431d-90c2-71b838f7e1e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4269370-x8n62\" (UID: \"d2356c8f-e080-431d-90c2-71b838f7e1e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4269370-x8n62"
Oct 30 20:24:04.801793 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:04.801719    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w959d\" (UniqueName: \"kubernetes.io/projected/d2356c8f-e080-431d-90c2-71b838f7e1e8-kube-api-access-w959d\") pod \"maas-api-revoke-sub-f4269370-x8n62\" (UID: \"d2356c8f-e080-431d-90c2-71b838f7e1e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4269370-x8n62"
Oct 30 20:24:04.801793 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:04.801740    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2356c8f-e080-431d-90c2-71b838f7e1e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4269370-x8n62\" (UID: \"d2356c8f-e080-431d-90c2-71b838f7e1e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4269370-x8n62"
Oct 30 20:24:04.903506 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:04.903474    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2356c8f-e080-431d-90c2-71b838f7e1e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4269370-x8n62\" (UID: \"d2356c8f-e080-431d-90c2-71b838f7e1e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4269370-x8n62"
Oct 30 20:24:04.903648 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:04.903540    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w959d\" (UniqueName: \"kubernetes.io/projected/d2356c8f-e080-431d-90c2-71b838f7e1e8-kube-api-access-w959d\") pod \"maas-api-revoke-sub-f4269370-x8n62\" (UID: \"d2356c8f-e080-431d-90c2-71b838f7e1e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4269370-x8n62"
Oct 30 20:24:04.903648 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:04.903558    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2356c8f-e080-431d-90c2-71b838f7e1e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4269370-x8n62\" (UID: \"d2356c8f-e080-431d-90c2-71b838f7e1e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4269370-x8n62"
Oct 30 20:24:04.904105 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:04.904078    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2356c8f-e080-431d-90c2-71b838f7e1e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4269370-x8n62\" (UID: \"d2356c8f-e080-431d-90c2-71b838f7e1e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4269370-x8n62"
Oct 30 20:24:04.905728 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:04.905705    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2356c8f-e080-431d-90c2-71b838f7e1e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4269370-x8n62\" (UID: \"d2356c8f-e080-431d-90c2-71b838f7e1e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4269370-x8n62"
Oct 30 20:24:04.919879 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:04.919846    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a404308-mqk8z"]
Oct 30 20:24:04.922024 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:04.921984    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6" podStartSLOduration=1.921973213 podStartE2EDuration="1.921973213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:24:04.921910928 +0000 UTC m=+2483.654126317" watchObservedRunningTime="2026-10-30 20:24:04.921973213 +0000 UTC m=+2483.654188603"
Oct 30 20:24:04.924508 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:04.924489    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a404308-mqk8z"
Oct 30 20:24:04.964970 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:04.964938    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w959d\" (UniqueName: \"kubernetes.io/projected/d2356c8f-e080-431d-90c2-71b838f7e1e8-kube-api-access-w959d\") pod \"maas-api-revoke-sub-f4269370-x8n62\" (UID: \"d2356c8f-e080-431d-90c2-71b838f7e1e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4269370-x8n62"
Oct 30 20:24:04.982886 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:04.982860    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a404308-mqk8z"]
Oct 30 20:24:05.000350 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:05.000327    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4269370-x8n62"
Oct 30 20:24:05.004887 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:05.004862    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bea5d8-aef0-4bf0-a76d-a584ce0088a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a404308-mqk8z\" (UID: \"f9bea5d8-aef0-4bf0-a76d-a584ce0088a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a404308-mqk8z"
Oct 30 20:24:05.005009 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:05.004903    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bea5d8-aef0-4bf0-a76d-a584ce0088a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a404308-mqk8z\" (UID: \"f9bea5d8-aef0-4bf0-a76d-a584ce0088a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a404308-mqk8z"
Oct 30 20:24:05.005104 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:05.005077    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whjsv\" (UniqueName: \"kubernetes.io/projected/f9bea5d8-aef0-4bf0-a76d-a584ce0088a9-kube-api-access-whjsv\") pod \"maas-api-revoke-sub-2a404308-mqk8z\" (UID: \"f9bea5d8-aef0-4bf0-a76d-a584ce0088a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a404308-mqk8z"
Oct 30 20:24:05.106072 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:05.106041    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bea5d8-aef0-4bf0-a76d-a584ce0088a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a404308-mqk8z\" (UID: \"f9bea5d8-aef0-4bf0-a76d-a584ce0088a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a404308-mqk8z"
Oct 30 20:24:05.106229 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:05.106212    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whjsv\" (UniqueName: \"kubernetes.io/projected/f9bea5d8-aef0-4bf0-a76d-a584ce0088a9-kube-api-access-whjsv\") pod \"maas-api-revoke-sub-2a404308-mqk8z\" (UID: \"f9bea5d8-aef0-4bf0-a76d-a584ce0088a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a404308-mqk8z"
Oct 30 20:24:05.106364 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:05.106351    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bea5d8-aef0-4bf0-a76d-a584ce0088a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a404308-mqk8z\" (UID: \"f9bea5d8-aef0-4bf0-a76d-a584ce0088a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a404308-mqk8z"
Oct 30 20:24:05.106727 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:05.106699    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bea5d8-aef0-4bf0-a76d-a584ce0088a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a404308-mqk8z\" (UID: \"f9bea5d8-aef0-4bf0-a76d-a584ce0088a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a404308-mqk8z"
Oct 30 20:24:05.108878 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:05.108856    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bea5d8-aef0-4bf0-a76d-a584ce0088a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a404308-mqk8z\" (UID: \"f9bea5d8-aef0-4bf0-a76d-a584ce0088a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a404308-mqk8z"
Oct 30 20:24:05.169192 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:05.169162    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whjsv\" (UniqueName: \"kubernetes.io/projected/f9bea5d8-aef0-4bf0-a76d-a584ce0088a9-kube-api-access-whjsv\") pod \"maas-api-revoke-sub-2a404308-mqk8z\" (UID: \"f9bea5d8-aef0-4bf0-a76d-a584ce0088a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a404308-mqk8z"
Oct 30 20:24:05.241656 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:05.241599    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a404308-mqk8z"
Oct 30 20:24:05.276621 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:05.276587    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb" podUID="6bdc83b4-72fc-42b0-83f9-740a2ce1cb71" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:24:05.277013 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:05.276991    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp" podUID="fbd4cbac-918f-4517-bdb7-9c4fcecf2401" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:24:05.393544 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:24:05.393511    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2356c8f_e080_431d_90c2_71b838f7e1e8.slice/crio-1640b09b3f5963664760dd030f19a9776cdaa1fd6fd540520a3b7944e2c94dc8 WatchSource:0}: Error finding container 1640b09b3f5963664760dd030f19a9776cdaa1fd6fd540520a3b7944e2c94dc8: Status 404 returned error can't find the container with id 1640b09b3f5963664760dd030f19a9776cdaa1fd6fd540520a3b7944e2c94dc8
Oct 30 20:24:05.399960 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:05.399933    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4269370-x8n62"]
Oct 30 20:24:05.595832 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:05.595804    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:24:05.615123 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:05.615100    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a404308-mqk8z"]
Oct 30 20:24:05.616209 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:24:05.616183    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9bea5d8_aef0_4bf0_a76d_a584ce0088a9.slice/crio-027615a0290f395653d905add4bf91aec6548abf3793d0fed3fd1c100d8fc16c WatchSource:0}: Error finding container 027615a0290f395653d905add4bf91aec6548abf3793d0fed3fd1c100d8fc16c: Status 404 returned error can't find the container with id 027615a0290f395653d905add4bf91aec6548abf3793d0fed3fd1c100d8fc16c
Oct 30 20:24:05.777836 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:05.777809    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a404308-mqk8z" event={"ID":"f9bea5d8-aef0-4bf0-a76d-a584ce0088a9","Type":"ContainerStarted","Data":"027615a0290f395653d905add4bf91aec6548abf3793d0fed3fd1c100d8fc16c"}
Oct 30 20:24:05.780104 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:05.780071    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4269370-x8n62" event={"ID":"d2356c8f-e080-431d-90c2-71b838f7e1e8","Type":"ContainerStarted","Data":"acb5f2f01397d1cd306f4e388314cd172aa87e90109bf716db761421670363a6"}
Oct 30 20:24:05.780205 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:05.780113    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4269370-x8n62" event={"ID":"d2356c8f-e080-431d-90c2-71b838f7e1e8","Type":"ContainerStarted","Data":"1640b09b3f5963664760dd030f19a9776cdaa1fd6fd540520a3b7944e2c94dc8"}
Oct 30 20:24:05.904306 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:05.904251    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4269370-x8n62" podStartSLOduration=1.904235435 podStartE2EDuration="1.904235435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:24:05.903713642 +0000 UTC m=+2484.635929033" watchObservedRunningTime="2026-10-30 20:24:05.904235435 +0000 UTC m=+2484.636450824"
Oct 30 20:24:06.605246 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:24:06.605213    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:24:06.787006 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:06.786932    5958 generic.go:356] "Generic (PLEG): container finished" podID="f9bea5d8-aef0-4bf0-a76d-a584ce0088a9" containerID="8d9a4279936677bab550c1ed60930c75a4df66c4bf5722bce068d675519732f9" exitCode=0
Oct 30 20:24:06.787366 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:06.787017    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a404308-mqk8z" event={"ID":"f9bea5d8-aef0-4bf0-a76d-a584ce0088a9","Type":"ContainerDied","Data":"8d9a4279936677bab550c1ed60930c75a4df66c4bf5722bce068d675519732f9"}
Oct 30 20:24:06.788904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:06.788873    5958 generic.go:356] "Generic (PLEG): container finished" podID="d2356c8f-e080-431d-90c2-71b838f7e1e8" containerID="acb5f2f01397d1cd306f4e388314cd172aa87e90109bf716db761421670363a6" exitCode=0
Oct 30 20:24:06.789020 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:06.788953    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4269370-x8n62" event={"ID":"d2356c8f-e080-431d-90c2-71b838f7e1e8","Type":"ContainerDied","Data":"acb5f2f01397d1cd306f4e388314cd172aa87e90109bf716db761421670363a6"}
Oct 30 20:24:07.941619 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:07.941252    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4269370-x8n62"
Oct 30 20:24:07.973172 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:07.973149    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a404308-mqk8z"
Oct 30 20:24:08.042884 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:08.042849    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2356c8f-e080-431d-90c2-71b838f7e1e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2356c8f-e080-431d-90c2-71b838f7e1e8-maas-api-openshift-ca\") pod \"d2356c8f-e080-431d-90c2-71b838f7e1e8\" (UID: \"d2356c8f-e080-431d-90c2-71b838f7e1e8\") "
Oct 30 20:24:08.043022 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:08.042912    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2356c8f-e080-431d-90c2-71b838f7e1e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2356c8f-e080-431d-90c2-71b838f7e1e8-maas-api-xks-ca\") pod \"d2356c8f-e080-431d-90c2-71b838f7e1e8\" (UID: \"d2356c8f-e080-431d-90c2-71b838f7e1e8\") "
Oct 30 20:24:08.043022 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:08.042930    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9bea5d8-aef0-4bf0-a76d-a584ce0088a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bea5d8-aef0-4bf0-a76d-a584ce0088a9-maas-api-openshift-ca\") pod \"f9bea5d8-aef0-4bf0-a76d-a584ce0088a9\" (UID: \"f9bea5d8-aef0-4bf0-a76d-a584ce0088a9\") "
Oct 30 20:24:08.043022 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:08.042981    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9bea5d8-aef0-4bf0-a76d-a584ce0088a9-kube-api-access-whjsv\" (UniqueName: \"kubernetes.io/projected/f9bea5d8-aef0-4bf0-a76d-a584ce0088a9-kube-api-access-whjsv\") pod \"f9bea5d8-aef0-4bf0-a76d-a584ce0088a9\" (UID: \"f9bea5d8-aef0-4bf0-a76d-a584ce0088a9\") "
Oct 30 20:24:08.043195 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:08.043030    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2356c8f-e080-431d-90c2-71b838f7e1e8-kube-api-access-w959d\" (UniqueName: \"kubernetes.io/projected/d2356c8f-e080-431d-90c2-71b838f7e1e8-kube-api-access-w959d\") pod \"d2356c8f-e080-431d-90c2-71b838f7e1e8\" (UID: \"d2356c8f-e080-431d-90c2-71b838f7e1e8\") "
Oct 30 20:24:08.043195 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:08.043093    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9bea5d8-aef0-4bf0-a76d-a584ce0088a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bea5d8-aef0-4bf0-a76d-a584ce0088a9-maas-api-xks-ca\") pod \"f9bea5d8-aef0-4bf0-a76d-a584ce0088a9\" (UID: \"f9bea5d8-aef0-4bf0-a76d-a584ce0088a9\") "
Oct 30 20:24:08.043301 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:08.043253    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2356c8f-e080-431d-90c2-71b838f7e1e8-maas-api-openshift-ca" pod "d2356c8f-e080-431d-90c2-71b838f7e1e8" (UID: "d2356c8f-e080-431d-90c2-71b838f7e1e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:24:08.043373 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:08.043350    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9bea5d8-aef0-4bf0-a76d-a584ce0088a9-maas-api-openshift-ca" pod "f9bea5d8-aef0-4bf0-a76d-a584ce0088a9" (UID: "f9bea5d8-aef0-4bf0-a76d-a584ce0088a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:24:08.043709 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:08.043691    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2356c8f-e080-431d-90c2-71b838f7e1e8-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:08.043804 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:08.043714    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bea5d8-aef0-4bf0-a76d-a584ce0088a9-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:08.045227 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:08.045160    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2356c8f-e080-431d-90c2-71b838f7e1e8-maas-api-xks-ca" pod "d2356c8f-e080-431d-90c2-71b838f7e1e8" (UID: "d2356c8f-e080-431d-90c2-71b838f7e1e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:24:08.045415 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:08.045399    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2356c8f-e080-431d-90c2-71b838f7e1e8-kube-api-access-w959d" pod "d2356c8f-e080-431d-90c2-71b838f7e1e8" (UID: "d2356c8f-e080-431d-90c2-71b838f7e1e8"). InnerVolumeSpecName "kube-api-access-w959d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:08.045807 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:08.045788    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9bea5d8-aef0-4bf0-a76d-a584ce0088a9-maas-api-xks-ca" pod "f9bea5d8-aef0-4bf0-a76d-a584ce0088a9" (UID: "f9bea5d8-aef0-4bf0-a76d-a584ce0088a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:24:08.045807 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:08.045791    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9bea5d8-aef0-4bf0-a76d-a584ce0088a9-kube-api-access-whjsv" pod "f9bea5d8-aef0-4bf0-a76d-a584ce0088a9" (UID: "f9bea5d8-aef0-4bf0-a76d-a584ce0088a9"). InnerVolumeSpecName "kube-api-access-whjsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:08.144866 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:08.144836    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2356c8f-e080-431d-90c2-71b838f7e1e8-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:08.144866 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:08.144866    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whjsv\" (UniqueName: \"kubernetes.io/projected/f9bea5d8-aef0-4bf0-a76d-a584ce0088a9-kube-api-access-whjsv\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:08.145025 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:08.144881    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w959d\" (UniqueName: \"kubernetes.io/projected/d2356c8f-e080-431d-90c2-71b838f7e1e8-kube-api-access-w959d\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:08.145025 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:08.144890    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bea5d8-aef0-4bf0-a76d-a584ce0088a9-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:08.802274 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:08.802244    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a404308-mqk8z"
Oct 30 20:24:08.802435 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:08.802290    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a404308-mqk8z" event={"ID":"f9bea5d8-aef0-4bf0-a76d-a584ce0088a9","Type":"ContainerDied","Data":"027615a0290f395653d905add4bf91aec6548abf3793d0fed3fd1c100d8fc16c"}
Oct 30 20:24:08.802435 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:08.802344    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="027615a0290f395653d905add4bf91aec6548abf3793d0fed3fd1c100d8fc16c"
Oct 30 20:24:08.803886 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:08.803857    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4269370-x8n62" event={"ID":"d2356c8f-e080-431d-90c2-71b838f7e1e8","Type":"ContainerDied","Data":"1640b09b3f5963664760dd030f19a9776cdaa1fd6fd540520a3b7944e2c94dc8"}
Oct 30 20:24:08.803886 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:08.803877    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4269370-x8n62"
Oct 30 20:24:08.804083 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:08.803903    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1640b09b3f5963664760dd030f19a9776cdaa1fd6fd540520a3b7944e2c94dc8"
Oct 30 20:24:11.977832 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:11.977792    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"]
Oct 30 20:24:11.978633 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:11.978611    5958 cpu_manager.go:414] "containerMap: removing container" podUID="d2356c8f-e080-431d-90c2-71b838f7e1e8" containerName="revoke-keys"
Oct 30 20:24:11.978633 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:11.978633    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2356c8f-e080-431d-90c2-71b838f7e1e8" containerName="revoke-keys"
Oct 30 20:24:11.978801 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:11.978685    5958 cpu_manager.go:414] "containerMap: removing container" podUID="f9bea5d8-aef0-4bf0-a76d-a584ce0088a9" containerName="revoke-keys"
Oct 30 20:24:11.978801 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:11.978693    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9bea5d8-aef0-4bf0-a76d-a584ce0088a9" containerName="revoke-keys"
Oct 30 20:24:11.978898 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:11.978848    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2356c8f-e080-431d-90c2-71b838f7e1e8" containerName="revoke-keys"
Oct 30 20:24:11.978898 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:11.978869    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9bea5d8-aef0-4bf0-a76d-a584ce0088a9" containerName="revoke-keys"
Oct 30 20:24:11.983361 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:11.983339    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.036173 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.036139    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-72db69-openshift-default-dockercfg-8kpkw\""
Oct 30 20:24:12.089157 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.089124    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"]
Oct 30 20:24:12.089774 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.089750    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-token\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.089835 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.089801    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-credential-socket\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.089933 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.089914    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b182630-ffc1-4770-a86c-3cdd5b763b92-istiod-ca-cert\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.089980 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.089952    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-workload-socket\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.089980 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.089969    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-podinfo\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.090076 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.089989    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-envoy\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.090165 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.090146    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd52l\" (UniqueName: \"kubernetes.io/projected/4b182630-ffc1-4770-a86c-3cdd5b763b92-kube-api-access-qd52l\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.090231 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.090186    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-data\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.090323 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.090310    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-workload-certs\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.191728 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.191687    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b182630-ffc1-4770-a86c-3cdd5b763b92-istiod-ca-cert\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.191920 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.191741    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-workload-socket\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.191920 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.191768    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-podinfo\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.191920 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.191795    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-envoy\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.192077 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.191931    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd52l\" (UniqueName: \"kubernetes.io/projected/4b182630-ffc1-4770-a86c-3cdd5b763b92-kube-api-access-qd52l\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.192077 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.191958    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-data\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.192077 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.192058    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-workload-certs\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.192236 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.192139    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-token\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.192236 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.192190    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-workload-socket\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.192236 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.192195    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-credential-socket\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.192412 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.192327    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-data\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.192570 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.192543    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-credential-socket\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.192694 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.192553    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b182630-ffc1-4770-a86c-3cdd5b763b92-istiod-ca-cert\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.192694 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.192566    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-workload-certs\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.194307 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.194283    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-envoy\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.194402 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.194384    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-podinfo\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.257641 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.257564    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-token\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.257921 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.257893    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd52l\" (UniqueName: \"kubernetes.io/projected/4b182630-ffc1-4770-a86c-3cdd5b763b92-kube-api-access-qd52l\") pod \"e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") " pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.300042 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.300016    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:12.730631 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.730588    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"]
Oct 30 20:24:12.767340 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:24:12.767299    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b182630_ffc1_4770_a86c_3cdd5b763b92.slice/crio-6c928e69b1c09080090a9ef67c6f4699d8532e5ea2f19daf54a1d026071c70b1 WatchSource:0}: Error finding container 6c928e69b1c09080090a9ef67c6f4699d8532e5ea2f19daf54a1d026071c70b1: Status 404 returned error can't find the container with id 6c928e69b1c09080090a9ef67c6f4699d8532e5ea2f19daf54a1d026071c70b1
Oct 30 20:24:12.769551 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.769513    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:24:12.769633 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.769586    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:24:12.769633 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.769620    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:24:12.830504 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:12.830478    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2" event={"ID":"4b182630-ffc1-4770-a86c-3cdd5b763b92","Type":"ContainerStarted","Data":"6c928e69b1c09080090a9ef67c6f4699d8532e5ea2f19daf54a1d026071c70b1"}
Oct 30 20:24:13.596410 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:13.596374    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:24:13.609464 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:13.609421    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:24:13.838353 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:13.838314    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2" event={"ID":"4b182630-ffc1-4770-a86c-3cdd5b763b92","Type":"ContainerStarted","Data":"fedb85bd1e7a0e8b19c9937cf3e9dc15567d32e16b9bac3ea89b765d2a36d329"}
Oct 30 20:24:13.849963 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:13.849904    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:24:13.961616 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:13.961561    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2" podStartSLOduration=2.961547566 podStartE2EDuration="2.961547566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:24:13.959122248 +0000 UTC m=+2492.691337640" watchObservedRunningTime="2026-10-30 20:24:13.961547566 +0000 UTC m=+2492.693763002"
Oct 30 20:24:14.300478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:14.300351    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:14.307171 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:14.307145    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:14.845136 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:14.845103    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:14.846803 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:14.846164    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:24:15.275690 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:15.275603    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp" podUID="fbd4cbac-918f-4517-bdb7-9c4fcecf2401" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:24:15.275846 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:15.275603    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb" podUID="6bdc83b4-72fc-42b0-83f9-740a2ce1cb71" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:24:16.043772 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.043748    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb"
Oct 30 20:24:16.140909 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.140881    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bdc83b4-72fc-42b0-83f9-740a2ce1cb71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bdc83b4-72fc-42b0-83f9-740a2ce1cb71-plugins-config-volume\") pod \"6bdc83b4-72fc-42b0-83f9-740a2ce1cb71\" (UID: \"6bdc83b4-72fc-42b0-83f9-740a2ce1cb71\") "
Oct 30 20:24:16.141636 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.140922    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bdc83b4-72fc-42b0-83f9-740a2ce1cb71-kube-api-access-m6l5q\" (UniqueName: \"kubernetes.io/projected/6bdc83b4-72fc-42b0-83f9-740a2ce1cb71-kube-api-access-m6l5q\") pod \"6bdc83b4-72fc-42b0-83f9-740a2ce1cb71\" (UID: \"6bdc83b4-72fc-42b0-83f9-740a2ce1cb71\") "
Oct 30 20:24:16.141636 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.141264    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bdc83b4-72fc-42b0-83f9-740a2ce1cb71-plugins-config-volume" pod "6bdc83b4-72fc-42b0-83f9-740a2ce1cb71" (UID: "6bdc83b4-72fc-42b0-83f9-740a2ce1cb71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:24:16.141636 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.141535    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bdc83b4-72fc-42b0-83f9-740a2ce1cb71-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:16.143422 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.143394    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bdc83b4-72fc-42b0-83f9-740a2ce1cb71-kube-api-access-m6l5q" pod "6bdc83b4-72fc-42b0-83f9-740a2ce1cb71" (UID: "6bdc83b4-72fc-42b0-83f9-740a2ce1cb71"). InnerVolumeSpecName "kube-api-access-m6l5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:16.147608 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.147589    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp"
Oct 30 20:24:16.242667 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.242631    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd4cbac-918f-4517-bdb7-9c4fcecf2401-kube-api-access-5w97d\" (UniqueName: \"kubernetes.io/projected/fbd4cbac-918f-4517-bdb7-9c4fcecf2401-kube-api-access-5w97d\") pod \"fbd4cbac-918f-4517-bdb7-9c4fcecf2401\" (UID: \"fbd4cbac-918f-4517-bdb7-9c4fcecf2401\") "
Oct 30 20:24:16.242848 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.242752    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd4cbac-918f-4517-bdb7-9c4fcecf2401-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd4cbac-918f-4517-bdb7-9c4fcecf2401-plugins-config-volume\") pod \"fbd4cbac-918f-4517-bdb7-9c4fcecf2401\" (UID: \"fbd4cbac-918f-4517-bdb7-9c4fcecf2401\") "
Oct 30 20:24:16.243175 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.243152    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd4cbac-918f-4517-bdb7-9c4fcecf2401-plugins-config-volume" pod "fbd4cbac-918f-4517-bdb7-9c4fcecf2401" (UID: "fbd4cbac-918f-4517-bdb7-9c4fcecf2401"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:24:16.243343 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.243320    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd4cbac-918f-4517-bdb7-9c4fcecf2401-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:16.243391 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.243348    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6l5q\" (UniqueName: \"kubernetes.io/projected/6bdc83b4-72fc-42b0-83f9-740a2ce1cb71-kube-api-access-m6l5q\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:16.244894 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.244873    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd4cbac-918f-4517-bdb7-9c4fcecf2401-kube-api-access-5w97d" pod "fbd4cbac-918f-4517-bdb7-9c4fcecf2401" (UID: "fbd4cbac-918f-4517-bdb7-9c4fcecf2401"). InnerVolumeSpecName "kube-api-access-5w97d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:16.344118 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.344082    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w97d\" (UniqueName: \"kubernetes.io/projected/fbd4cbac-918f-4517-bdb7-9c4fcecf2401-kube-api-access-5w97d\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:16.859688 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.859650    5958 generic.go:356] "Generic (PLEG): container finished" podID="fbd4cbac-918f-4517-bdb7-9c4fcecf2401" containerID="44bf9565b02e5439fd7ffebcbab406f1c86a902e4f13789625c21bb621d1dbdf" exitCode=0
Oct 30 20:24:16.859849 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.859709    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp"
Oct 30 20:24:16.859849 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.859725    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp" event={"ID":"fbd4cbac-918f-4517-bdb7-9c4fcecf2401","Type":"ContainerDied","Data":"44bf9565b02e5439fd7ffebcbab406f1c86a902e4f13789625c21bb621d1dbdf"}
Oct 30 20:24:16.859849 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.859807    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp" event={"ID":"fbd4cbac-918f-4517-bdb7-9c4fcecf2401","Type":"ContainerDied","Data":"729935eabc3f5eab930874fdc59402813207d630327ea8a9b36abb512e018f86"}
Oct 30 20:24:16.859849 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.859832    5958 scope.go:122] "RemoveContainer" containerID="44bf9565b02e5439fd7ffebcbab406f1c86a902e4f13789625c21bb621d1dbdf"
Oct 30 20:24:16.861524 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.861500    5958 generic.go:356] "Generic (PLEG): container finished" podID="6bdc83b4-72fc-42b0-83f9-740a2ce1cb71" containerID="c0e6d5c2c21b5c54b27450229391a7a70639939d1862f2a82cb4ac8d7bc5c733" exitCode=0
Oct 30 20:24:16.861642 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.861570    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb"
Oct 30 20:24:16.861642 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.861585    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb" event={"ID":"6bdc83b4-72fc-42b0-83f9-740a2ce1cb71","Type":"ContainerDied","Data":"c0e6d5c2c21b5c54b27450229391a7a70639939d1862f2a82cb4ac8d7bc5c733"}
Oct 30 20:24:16.861642 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.861635    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb" event={"ID":"6bdc83b4-72fc-42b0-83f9-740a2ce1cb71","Type":"ContainerDied","Data":"b7cbdbc452936a0077674e30b226c08e2197bec61f07ae19481f6b429fea1155"}
Oct 30 20:24:16.870578 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.870557    5958 scope.go:122] "RemoveContainer" containerID="44bf9565b02e5439fd7ffebcbab406f1c86a902e4f13789625c21bb621d1dbdf"
Oct 30 20:24:16.870898 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:24:16.870806    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44bf9565b02e5439fd7ffebcbab406f1c86a902e4f13789625c21bb621d1dbdf\": container with ID starting with 44bf9565b02e5439fd7ffebcbab406f1c86a902e4f13789625c21bb621d1dbdf not found: ID does not exist" containerID="44bf9565b02e5439fd7ffebcbab406f1c86a902e4f13789625c21bb621d1dbdf"
Oct 30 20:24:16.870898 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.870829    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44bf9565b02e5439fd7ffebcbab406f1c86a902e4f13789625c21bb621d1dbdf"} err="failed to get container status \"44bf9565b02e5439fd7ffebcbab406f1c86a902e4f13789625c21bb621d1dbdf\": rpc error: code = NotFound desc = could not find container \"44bf9565b02e5439fd7ffebcbab406f1c86a902e4f13789625c21bb621d1dbdf\": container with ID starting with 44bf9565b02e5439fd7ffebcbab406f1c86a902e4f13789625c21bb621d1dbdf not found: ID does not exist"
Oct 30 20:24:16.870898 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.870845    5958 scope.go:122] "RemoveContainer" containerID="c0e6d5c2c21b5c54b27450229391a7a70639939d1862f2a82cb4ac8d7bc5c733"
Oct 30 20:24:16.880529 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.880320    5958 scope.go:122] "RemoveContainer" containerID="c0e6d5c2c21b5c54b27450229391a7a70639939d1862f2a82cb4ac8d7bc5c733"
Oct 30 20:24:16.880659 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:24:16.880636    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0e6d5c2c21b5c54b27450229391a7a70639939d1862f2a82cb4ac8d7bc5c733\": container with ID starting with c0e6d5c2c21b5c54b27450229391a7a70639939d1862f2a82cb4ac8d7bc5c733 not found: ID does not exist" containerID="c0e6d5c2c21b5c54b27450229391a7a70639939d1862f2a82cb4ac8d7bc5c733"
Oct 30 20:24:16.880703 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:16.880667    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0e6d5c2c21b5c54b27450229391a7a70639939d1862f2a82cb4ac8d7bc5c733"} err="failed to get container status \"c0e6d5c2c21b5c54b27450229391a7a70639939d1862f2a82cb4ac8d7bc5c733\": rpc error: code = NotFound desc = could not find container \"c0e6d5c2c21b5c54b27450229391a7a70639939d1862f2a82cb4ac8d7bc5c733\": container with ID starting with c0e6d5c2c21b5c54b27450229391a7a70639939d1862f2a82cb4ac8d7bc5c733 not found: ID does not exist"
Oct 30 20:24:17.031382 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:17.031298    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp"]
Oct 30 20:24:17.080967 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:17.080931    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-f75272-57c5f7755f-xkhwp"]
Oct 30 20:24:17.144880 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:17.144847    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb"]
Oct 30 20:24:17.154375 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:17.154341    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-f75272-f98d77bdd-67gbb"]
Oct 30 20:24:17.400191 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:17.400149    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bdc83b4-72fc-42b0-83f9-740a2ce1cb71" path="/var/lib/kubelet/pods/6bdc83b4-72fc-42b0-83f9-740a2ce1cb71/volumes"
Oct 30 20:24:17.400682 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:17.400659    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbd4cbac-918f-4517-bdb7-9c4fcecf2401" path="/var/lib/kubelet/pods/fbd4cbac-918f-4517-bdb7-9c4fcecf2401/volumes"
Oct 30 20:24:20.813474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:20.810272    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf"]
Oct 30 20:24:20.813474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:20.811821    5958 cpu_manager.go:414] "containerMap: removing container" podUID="6bdc83b4-72fc-42b0-83f9-740a2ce1cb71" containerName="payload-processing"
Oct 30 20:24:20.813474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:20.811846    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bdc83b4-72fc-42b0-83f9-740a2ce1cb71" containerName="payload-processing"
Oct 30 20:24:20.813474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:20.811863    5958 cpu_manager.go:414] "containerMap: removing container" podUID="fbd4cbac-918f-4517-bdb7-9c4fcecf2401" containerName="payload-pre-processing"
Oct 30 20:24:20.813474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:20.811871    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd4cbac-918f-4517-bdb7-9c4fcecf2401" containerName="payload-pre-processing"
Oct 30 20:24:20.813474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:20.812002    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bdc83b4-72fc-42b0-83f9-740a2ce1cb71" containerName="payload-processing"
Oct 30 20:24:20.813474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:20.812014    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd4cbac-918f-4517-bdb7-9c4fcecf2401" containerName="payload-pre-processing"
Oct 30 20:24:20.882681 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:20.882647    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf"
Oct 30 20:24:20.897972 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:20.897921    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf"]
Oct 30 20:24:20.944909 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:20.944884    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-72db69\""
Oct 30 20:24:20.994754 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:20.994726    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-72db69-dockercfg-hxrm4\""
Oct 30 20:24:20.997904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:20.997877    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4p55\" (UniqueName: \"kubernetes.io/projected/3e4039e9-32bf-4575-aba9-db44bb0c5077-kube-api-access-g4p55\") pod \"payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf\" (UID: \"3e4039e9-32bf-4575-aba9-db44bb0c5077\") " pod="openshift-ingress/payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf"
Oct 30 20:24:20.998095 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:20.998075    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e4039e9-32bf-4575-aba9-db44bb0c5077-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf\" (UID: \"3e4039e9-32bf-4575-aba9-db44bb0c5077\") " pod="openshift-ingress/payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf"
Oct 30 20:24:21.099379 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:21.099299    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e4039e9-32bf-4575-aba9-db44bb0c5077-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf\" (UID: \"3e4039e9-32bf-4575-aba9-db44bb0c5077\") " pod="openshift-ingress/payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf"
Oct 30 20:24:21.099557 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:21.099424    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4p55\" (UniqueName: \"kubernetes.io/projected/3e4039e9-32bf-4575-aba9-db44bb0c5077-kube-api-access-g4p55\") pod \"payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf\" (UID: \"3e4039e9-32bf-4575-aba9-db44bb0c5077\") " pod="openshift-ingress/payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf"
Oct 30 20:24:21.100058 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:21.100037    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e4039e9-32bf-4575-aba9-db44bb0c5077-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf\" (UID: \"3e4039e9-32bf-4575-aba9-db44bb0c5077\") " pod="openshift-ingress/payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf"
Oct 30 20:24:21.163638 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:21.163608    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4p55\" (UniqueName: \"kubernetes.io/projected/3e4039e9-32bf-4575-aba9-db44bb0c5077-kube-api-access-g4p55\") pod \"payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf\" (UID: \"3e4039e9-32bf-4575-aba9-db44bb0c5077\") " pod="openshift-ingress/payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf"
Oct 30 20:24:21.217749 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:21.217684    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf"
Oct 30 20:24:21.664708 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:21.664680    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf"]
Oct 30 20:24:21.666215 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:24:21.666177    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e4039e9_32bf_4575_aba9_db44bb0c5077.slice/crio-279bd80050491ee4309d165e5733c68ef644d579543c3945d2a555bb85616f24 WatchSource:0}: Error finding container 279bd80050491ee4309d165e5733c68ef644d579543c3945d2a555bb85616f24: Status 404 returned error can't find the container with id 279bd80050491ee4309d165e5733c68ef644d579543c3945d2a555bb85616f24
Oct 30 20:24:21.914697 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:21.914607    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf" event={"ID":"3e4039e9-32bf-4575-aba9-db44bb0c5077","Type":"ContainerStarted","Data":"4a0db6334aad8dd2b8fe81e42402eba2e7c74ba78edf66b0fb6e03ef3aeb9894"}
Oct 30 20:24:21.914697 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:21.914662    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf" event={"ID":"3e4039e9-32bf-4575-aba9-db44bb0c5077","Type":"ContainerStarted","Data":"279bd80050491ee4309d165e5733c68ef644d579543c3945d2a555bb85616f24"}
Oct 30 20:24:21.915124 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:21.914706    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf"
Oct 30 20:24:22.032791 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:22.032730    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf" podStartSLOduration=2.032711361 podStartE2EDuration="2.032711361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:24:22.031893425 +0000 UTC m=+2500.764108816" watchObservedRunningTime="2026-10-30 20:24:22.032711361 +0000 UTC m=+2500.764926746"
Oct 30 20:24:32.924338 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:32.924305    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf"
Oct 30 20:24:36.606743 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:24:36.606693    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:24:52.296226 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:52.296152    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z"]
Oct 30 20:24:52.296706 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:52.296353    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z" podUID="2836bfd6-8a39-4194-8ac7-723940c07a2e" containerName="maas-api" containerID="cri-o://00c42a3bab2be68efc4789676a87de8972fdc5e4ada9878fc454048dd9e13415" gracePeriod=30
Oct 30 20:24:52.567590 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:52.567560    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z"
Oct 30 20:24:52.708817 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:52.708668    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2836bfd6-8a39-4194-8ac7-723940c07a2e-kube-api-access-js2ws\" (UniqueName: \"kubernetes.io/projected/2836bfd6-8a39-4194-8ac7-723940c07a2e-kube-api-access-js2ws\") pod \"2836bfd6-8a39-4194-8ac7-723940c07a2e\" (UID: \"2836bfd6-8a39-4194-8ac7-723940c07a2e\") "
Oct 30 20:24:52.708817 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:52.708767    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2836bfd6-8a39-4194-8ac7-723940c07a2e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2836bfd6-8a39-4194-8ac7-723940c07a2e-maas-api-tls\") pod \"2836bfd6-8a39-4194-8ac7-723940c07a2e\" (UID: \"2836bfd6-8a39-4194-8ac7-723940c07a2e\") "
Oct 30 20:24:52.709055 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:52.708849    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2836bfd6-8a39-4194-8ac7-723940c07a2e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2836bfd6-8a39-4194-8ac7-723940c07a2e-openshift-service-ca\") pod \"2836bfd6-8a39-4194-8ac7-723940c07a2e\" (UID: \"2836bfd6-8a39-4194-8ac7-723940c07a2e\") "
Oct 30 20:24:52.709055 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:52.708924    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2836bfd6-8a39-4194-8ac7-723940c07a2e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2836bfd6-8a39-4194-8ac7-723940c07a2e-metrics-certs\") pod \"2836bfd6-8a39-4194-8ac7-723940c07a2e\" (UID: \"2836bfd6-8a39-4194-8ac7-723940c07a2e\") "
Oct 30 20:24:52.710954 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:52.710902    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2836bfd6-8a39-4194-8ac7-723940c07a2e-openshift-service-ca" pod "2836bfd6-8a39-4194-8ac7-723940c07a2e" (UID: "2836bfd6-8a39-4194-8ac7-723940c07a2e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:24:52.713561 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:52.713532    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2836bfd6-8a39-4194-8ac7-723940c07a2e-maas-api-tls" pod "2836bfd6-8a39-4194-8ac7-723940c07a2e" (UID: "2836bfd6-8a39-4194-8ac7-723940c07a2e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:24:52.713682 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:52.713582    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2836bfd6-8a39-4194-8ac7-723940c07a2e-kube-api-access-js2ws" pod "2836bfd6-8a39-4194-8ac7-723940c07a2e" (UID: "2836bfd6-8a39-4194-8ac7-723940c07a2e"). InnerVolumeSpecName "kube-api-access-js2ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:52.720058 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:52.720035    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2836bfd6-8a39-4194-8ac7-723940c07a2e-metrics-certs" pod "2836bfd6-8a39-4194-8ac7-723940c07a2e" (UID: "2836bfd6-8a39-4194-8ac7-723940c07a2e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:24:52.811119 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:52.811051    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2836bfd6-8a39-4194-8ac7-723940c07a2e-maas-api-tls\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:52.811119 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:52.811080    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2836bfd6-8a39-4194-8ac7-723940c07a2e-openshift-service-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:52.811119 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:52.811090    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2836bfd6-8a39-4194-8ac7-723940c07a2e-metrics-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:52.811119 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:52.811098    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js2ws\" (UniqueName: \"kubernetes.io/projected/2836bfd6-8a39-4194-8ac7-723940c07a2e-kube-api-access-js2ws\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:53.129341 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:53.129303    5958 generic.go:356] "Generic (PLEG): container finished" podID="2836bfd6-8a39-4194-8ac7-723940c07a2e" containerID="00c42a3bab2be68efc4789676a87de8972fdc5e4ada9878fc454048dd9e13415" exitCode=0
Oct 30 20:24:53.129555 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:53.129349    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z" event={"ID":"2836bfd6-8a39-4194-8ac7-723940c07a2e","Type":"ContainerDied","Data":"00c42a3bab2be68efc4789676a87de8972fdc5e4ada9878fc454048dd9e13415"}
Oct 30 20:24:53.129555 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:53.129424    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z" event={"ID":"2836bfd6-8a39-4194-8ac7-723940c07a2e","Type":"ContainerDied","Data":"6609203cd3efff75c5885fb43bde2115df76b2936c9ddd9fc991f112a216b604"}
Oct 30 20:24:53.129555 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:53.129489    5958 scope.go:122] "RemoveContainer" containerID="00c42a3bab2be68efc4789676a87de8972fdc5e4ada9878fc454048dd9e13415"
Oct 30 20:24:53.129742 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:53.129581    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z"
Oct 30 20:24:53.141423 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:53.141406    5958 scope.go:122] "RemoveContainer" containerID="00c42a3bab2be68efc4789676a87de8972fdc5e4ada9878fc454048dd9e13415"
Oct 30 20:24:53.141706 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:24:53.141688    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00c42a3bab2be68efc4789676a87de8972fdc5e4ada9878fc454048dd9e13415\": container with ID starting with 00c42a3bab2be68efc4789676a87de8972fdc5e4ada9878fc454048dd9e13415 not found: ID does not exist" containerID="00c42a3bab2be68efc4789676a87de8972fdc5e4ada9878fc454048dd9e13415"
Oct 30 20:24:53.141751 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:53.141714    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c42a3bab2be68efc4789676a87de8972fdc5e4ada9878fc454048dd9e13415"} err="failed to get container status \"00c42a3bab2be68efc4789676a87de8972fdc5e4ada9878fc454048dd9e13415\": rpc error: code = NotFound desc = could not find container \"00c42a3bab2be68efc4789676a87de8972fdc5e4ada9878fc454048dd9e13415\": container with ID starting with 00c42a3bab2be68efc4789676a87de8972fdc5e4ada9878fc454048dd9e13415 not found: ID does not exist"
Oct 30 20:24:53.296724 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:53.296687    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z"]
Oct 30 20:24:53.346875 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:53.346843    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bebb42-7cf8f55455-pz52z"]
Oct 30 20:24:53.399046 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:24:53.398978    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2836bfd6-8a39-4194-8ac7-723940c07a2e" path="/var/lib/kubelet/pods/2836bfd6-8a39-4194-8ac7-723940c07a2e/volumes"
Oct 30 20:25:00.836593 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:00.836561    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"]
Oct 30 20:25:00.836980 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:00.836774    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg" podUID="590c5cad-401a-49be-a4b1-79892dc39d5d" containerName="istio-proxy" containerID="cri-o://0376c7d0aae1c7f91cda1b39b9eb4e5dfb133fdaec604d26f6fc482ac89a8f0d" gracePeriod=30
Oct 30 20:25:03.137946 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:03.137900    5958 patch_prober.go:29] interesting pod/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:25:03.138340 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:03.137974    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg" podUID="590c5cad-401a-49be-a4b1-79892dc39d5d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:25:06.098723 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.098700    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 20:25:06.183680 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.183622    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-data\") pod \"590c5cad-401a-49be-a4b1-79892dc39d5d\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") "
Oct 30 20:25:06.183680 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.183662    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/590c5cad-401a-49be-a4b1-79892dc39d5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/590c5cad-401a-49be-a4b1-79892dc39d5d-istiod-ca-cert\") pod \"590c5cad-401a-49be-a4b1-79892dc39d5d\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") "
Oct 30 20:25:06.183871 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.183692    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-envoy\") pod \"590c5cad-401a-49be-a4b1-79892dc39d5d\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") "
Oct 30 20:25:06.183871 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.183742    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/590c5cad-401a-49be-a4b1-79892dc39d5d-kube-api-access-cm9nh\" (UniqueName: \"kubernetes.io/projected/590c5cad-401a-49be-a4b1-79892dc39d5d-kube-api-access-cm9nh\") pod \"590c5cad-401a-49be-a4b1-79892dc39d5d\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") "
Oct 30 20:25:06.183871 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.183792    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-workload-socket\") pod \"590c5cad-401a-49be-a4b1-79892dc39d5d\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") "
Oct 30 20:25:06.183871 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.183820    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-token\" (UniqueName: \"kubernetes.io/projected/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-token\") pod \"590c5cad-401a-49be-a4b1-79892dc39d5d\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") "
Oct 30 20:25:06.183871 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.183863    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-workload-certs\") pod \"590c5cad-401a-49be-a4b1-79892dc39d5d\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") "
Oct 30 20:25:06.184124 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.183895    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-credential-socket\") pod \"590c5cad-401a-49be-a4b1-79892dc39d5d\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") "
Oct 30 20:25:06.184124 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.183921    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-data" pod "590c5cad-401a-49be-a4b1-79892dc39d5d" (UID: "590c5cad-401a-49be-a4b1-79892dc39d5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:06.184124 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.183931    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-podinfo\") pod \"590c5cad-401a-49be-a4b1-79892dc39d5d\" (UID: \"590c5cad-401a-49be-a4b1-79892dc39d5d\") "
Oct 30 20:25:06.184262 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.184030    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/590c5cad-401a-49be-a4b1-79892dc39d5d-istiod-ca-cert" pod "590c5cad-401a-49be-a4b1-79892dc39d5d" (UID: "590c5cad-401a-49be-a4b1-79892dc39d5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:25:06.184262 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.184134    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-workload-socket" pod "590c5cad-401a-49be-a4b1-79892dc39d5d" (UID: "590c5cad-401a-49be-a4b1-79892dc39d5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:06.184358 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.184260    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-workload-certs" pod "590c5cad-401a-49be-a4b1-79892dc39d5d" (UID: "590c5cad-401a-49be-a4b1-79892dc39d5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:06.184358 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.184284    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-credential-socket" pod "590c5cad-401a-49be-a4b1-79892dc39d5d" (UID: "590c5cad-401a-49be-a4b1-79892dc39d5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:06.184670 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.184647    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:06.184761 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.184677    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:06.184761 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.184694    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:06.184761 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.184706    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:06.184761 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.184719    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/590c5cad-401a-49be-a4b1-79892dc39d5d-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:06.185962 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.185944    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-envoy" pod "590c5cad-401a-49be-a4b1-79892dc39d5d" (UID: "590c5cad-401a-49be-a4b1-79892dc39d5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:06.186385 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.186366    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-token" pod "590c5cad-401a-49be-a4b1-79892dc39d5d" (UID: "590c5cad-401a-49be-a4b1-79892dc39d5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:06.186571 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.186553    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-podinfo" pod "590c5cad-401a-49be-a4b1-79892dc39d5d" (UID: "590c5cad-401a-49be-a4b1-79892dc39d5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:25:06.186895 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.186569    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590c5cad-401a-49be-a4b1-79892dc39d5d-kube-api-access-cm9nh" pod "590c5cad-401a-49be-a4b1-79892dc39d5d" (UID: "590c5cad-401a-49be-a4b1-79892dc39d5d"). InnerVolumeSpecName "kube-api-access-cm9nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:06.210163 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.210139    5958 generic.go:356] "Generic (PLEG): container finished" podID="590c5cad-401a-49be-a4b1-79892dc39d5d" containerID="0376c7d0aae1c7f91cda1b39b9eb4e5dfb133fdaec604d26f6fc482ac89a8f0d" exitCode=0
Oct 30 20:25:06.210251 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.210193    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"
Oct 30 20:25:06.210251 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.210213    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg" event={"ID":"590c5cad-401a-49be-a4b1-79892dc39d5d","Type":"ContainerDied","Data":"0376c7d0aae1c7f91cda1b39b9eb4e5dfb133fdaec604d26f6fc482ac89a8f0d"}
Oct 30 20:25:06.210337 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.210265    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg" event={"ID":"590c5cad-401a-49be-a4b1-79892dc39d5d","Type":"ContainerDied","Data":"4b0fdbc5ed87f86154965f9dd1b7f0e15d5255352127833acf26851224910501"}
Oct 30 20:25:06.210337 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.210286    5958 scope.go:122] "RemoveContainer" containerID="0376c7d0aae1c7f91cda1b39b9eb4e5dfb133fdaec604d26f6fc482ac89a8f0d"
Oct 30 20:25:06.220816 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.220797    5958 scope.go:122] "RemoveContainer" containerID="44b5342dca4dbe6d31592852da154980e2c82c0088a84e20ba8a998573a55bcc"
Oct 30 20:25:06.229574 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.229556    5958 scope.go:122] "RemoveContainer" containerID="0376c7d0aae1c7f91cda1b39b9eb4e5dfb133fdaec604d26f6fc482ac89a8f0d"
Oct 30 20:25:06.229810 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:25:06.229795    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0376c7d0aae1c7f91cda1b39b9eb4e5dfb133fdaec604d26f6fc482ac89a8f0d\": container with ID starting with 0376c7d0aae1c7f91cda1b39b9eb4e5dfb133fdaec604d26f6fc482ac89a8f0d not found: ID does not exist" containerID="0376c7d0aae1c7f91cda1b39b9eb4e5dfb133fdaec604d26f6fc482ac89a8f0d"
Oct 30 20:25:06.229877 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.229816    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0376c7d0aae1c7f91cda1b39b9eb4e5dfb133fdaec604d26f6fc482ac89a8f0d"} err="failed to get container status \"0376c7d0aae1c7f91cda1b39b9eb4e5dfb133fdaec604d26f6fc482ac89a8f0d\": rpc error: code = NotFound desc = could not find container \"0376c7d0aae1c7f91cda1b39b9eb4e5dfb133fdaec604d26f6fc482ac89a8f0d\": container with ID starting with 0376c7d0aae1c7f91cda1b39b9eb4e5dfb133fdaec604d26f6fc482ac89a8f0d not found: ID does not exist"
Oct 30 20:25:06.229877 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.229831    5958 scope.go:122] "RemoveContainer" containerID="44b5342dca4dbe6d31592852da154980e2c82c0088a84e20ba8a998573a55bcc"
Oct 30 20:25:06.230066 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:25:06.230053    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44b5342dca4dbe6d31592852da154980e2c82c0088a84e20ba8a998573a55bcc\": container with ID starting with 44b5342dca4dbe6d31592852da154980e2c82c0088a84e20ba8a998573a55bcc not found: ID does not exist" containerID="44b5342dca4dbe6d31592852da154980e2c82c0088a84e20ba8a998573a55bcc"
Oct 30 20:25:06.230106 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.230068    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44b5342dca4dbe6d31592852da154980e2c82c0088a84e20ba8a998573a55bcc"} err="failed to get container status \"44b5342dca4dbe6d31592852da154980e2c82c0088a84e20ba8a998573a55bcc\": rpc error: code = NotFound desc = could not find container \"44b5342dca4dbe6d31592852da154980e2c82c0088a84e20ba8a998573a55bcc\": container with ID starting with 44b5342dca4dbe6d31592852da154980e2c82c0088a84e20ba8a998573a55bcc not found: ID does not exist"
Oct 30 20:25:06.286033 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.285996    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:06.286033 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.286034    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm9nh\" (UniqueName: \"kubernetes.io/projected/590c5cad-401a-49be-a4b1-79892dc39d5d-kube-api-access-cm9nh\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:06.286230 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.286050    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:06.286230 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.286065    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/590c5cad-401a-49be-a4b1-79892dc39d5d-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:06.388824 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.388794    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"]
Oct 30 20:25:06.392921 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:06.392895    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f75272-openshift-default-649458586c-cj4gg"]
Oct 30 20:25:06.616367 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:25:06.616332    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:25:07.406619 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:07.401061    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="590c5cad-401a-49be-a4b1-79892dc39d5d" path="/var/lib/kubelet/pods/590c5cad-401a-49be-a4b1-79892dc39d5d/volumes"
Oct 30 20:25:18.075345 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.075298    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"]
Oct 30 20:25:18.075984 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.075964    5958 cpu_manager.go:414] "containerMap: removing container" podUID="590c5cad-401a-49be-a4b1-79892dc39d5d" containerName="istio-proxy"
Oct 30 20:25:18.076035 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.075987    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="590c5cad-401a-49be-a4b1-79892dc39d5d" containerName="istio-proxy"
Oct 30 20:25:18.076035 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.076020    5958 cpu_manager.go:414] "containerMap: removing container" podUID="2836bfd6-8a39-4194-8ac7-723940c07a2e" containerName="maas-api"
Oct 30 20:25:18.076035 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.076025    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2836bfd6-8a39-4194-8ac7-723940c07a2e" containerName="maas-api"
Oct 30 20:25:18.076151 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.076140    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="2836bfd6-8a39-4194-8ac7-723940c07a2e" containerName="maas-api"
Oct 30 20:25:18.076186 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.076157    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="590c5cad-401a-49be-a4b1-79892dc39d5d" containerName="istio-proxy"
Oct 30 20:25:18.076186 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.076167    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="590c5cad-401a-49be-a4b1-79892dc39d5d" containerName="istio-proxy"
Oct 30 20:25:18.079898 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.079870    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:25:18.142524 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.142490    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"]
Oct 30 20:25:18.184605 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.184561    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-01e8dc\"/\"kube-root-ca.crt\""
Oct 30 20:25:18.184819 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.184672    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-01e8dc\"/\"openshift-service-ca.crt\""
Oct 30 20:25:18.184819 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.184694    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-01e8dc\"/\"default-dockercfg-8kwzs\""
Oct 30 20:25:18.187256 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.187223    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-01e8dc\"/\"test-model-01e8dc-kserve-self-signed-certs\""
Oct 30 20:25:18.232977 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.232932    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-tmp-dir\") pod \"test-model-01e8dc-kserve-f8f46c89c-2ctnj\" (UID: \"b32f150b-7845-48de-8c12-61a7c1126976\") " pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:25:18.233146 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.232980    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-dshm\") pod \"test-model-01e8dc-kserve-f8f46c89c-2ctnj\" (UID: \"b32f150b-7845-48de-8c12-61a7c1126976\") " pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:25:18.233146 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.233079    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx8hs\" (UniqueName: \"kubernetes.io/projected/b32f150b-7845-48de-8c12-61a7c1126976-kube-api-access-vx8hs\") pod \"test-model-01e8dc-kserve-f8f46c89c-2ctnj\" (UID: \"b32f150b-7845-48de-8c12-61a7c1126976\") " pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:25:18.233261 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.233241    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-model-cache\") pod \"test-model-01e8dc-kserve-f8f46c89c-2ctnj\" (UID: \"b32f150b-7845-48de-8c12-61a7c1126976\") " pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:25:18.233303 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.233277    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b32f150b-7845-48de-8c12-61a7c1126976-tls-certs\") pod \"test-model-01e8dc-kserve-f8f46c89c-2ctnj\" (UID: \"b32f150b-7845-48de-8c12-61a7c1126976\") " pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:25:18.233424 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.233309    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-home\") pod \"test-model-01e8dc-kserve-f8f46c89c-2ctnj\" (UID: \"b32f150b-7845-48de-8c12-61a7c1126976\") " pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:25:18.335017 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.334947    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-model-cache\") pod \"test-model-01e8dc-kserve-f8f46c89c-2ctnj\" (UID: \"b32f150b-7845-48de-8c12-61a7c1126976\") " pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:25:18.335017 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.334986    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b32f150b-7845-48de-8c12-61a7c1126976-tls-certs\") pod \"test-model-01e8dc-kserve-f8f46c89c-2ctnj\" (UID: \"b32f150b-7845-48de-8c12-61a7c1126976\") " pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:25:18.335224 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.335109    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-home\") pod \"test-model-01e8dc-kserve-f8f46c89c-2ctnj\" (UID: \"b32f150b-7845-48de-8c12-61a7c1126976\") " pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:25:18.335279 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.335260    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-tmp-dir\") pod \"test-model-01e8dc-kserve-f8f46c89c-2ctnj\" (UID: \"b32f150b-7845-48de-8c12-61a7c1126976\") " pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:25:18.335347 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.335289    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-dshm\") pod \"test-model-01e8dc-kserve-f8f46c89c-2ctnj\" (UID: \"b32f150b-7845-48de-8c12-61a7c1126976\") " pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:25:18.335347 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.335304    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-model-cache\") pod \"test-model-01e8dc-kserve-f8f46c89c-2ctnj\" (UID: \"b32f150b-7845-48de-8c12-61a7c1126976\") " pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:25:18.335347 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.335336    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx8hs\" (UniqueName: \"kubernetes.io/projected/b32f150b-7845-48de-8c12-61a7c1126976-kube-api-access-vx8hs\") pod \"test-model-01e8dc-kserve-f8f46c89c-2ctnj\" (UID: \"b32f150b-7845-48de-8c12-61a7c1126976\") " pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:25:18.335512 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.335413    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-home\") pod \"test-model-01e8dc-kserve-f8f46c89c-2ctnj\" (UID: \"b32f150b-7845-48de-8c12-61a7c1126976\") " pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:25:18.335620 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.335602    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-tmp-dir\") pod \"test-model-01e8dc-kserve-f8f46c89c-2ctnj\" (UID: \"b32f150b-7845-48de-8c12-61a7c1126976\") " pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:25:18.337371 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.337351    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-dshm\") pod \"test-model-01e8dc-kserve-f8f46c89c-2ctnj\" (UID: \"b32f150b-7845-48de-8c12-61a7c1126976\") " pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:25:18.337484 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.337425    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b32f150b-7845-48de-8c12-61a7c1126976-tls-certs\") pod \"test-model-01e8dc-kserve-f8f46c89c-2ctnj\" (UID: \"b32f150b-7845-48de-8c12-61a7c1126976\") " pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:25:18.398802 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.398770    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx8hs\" (UniqueName: \"kubernetes.io/projected/b32f150b-7845-48de-8c12-61a7c1126976-kube-api-access-vx8hs\") pod \"test-model-01e8dc-kserve-f8f46c89c-2ctnj\" (UID: \"b32f150b-7845-48de-8c12-61a7c1126976\") " pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:25:18.453796 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.453763    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:25:18.833493 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:18.833413    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"]
Oct 30 20:25:18.834363 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:25:18.834342    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb32f150b_7845_48de_8c12_61a7c1126976.slice/crio-4ca6d0351c8e6ac9c1f881fb55c12353faa95298ca134f4016e161ba4ded8bb7 WatchSource:0}: Error finding container 4ca6d0351c8e6ac9c1f881fb55c12353faa95298ca134f4016e161ba4ded8bb7: Status 404 returned error can't find the container with id 4ca6d0351c8e6ac9c1f881fb55c12353faa95298ca134f4016e161ba4ded8bb7
Oct 30 20:25:19.295269 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:19.295171    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj" event={"ID":"b32f150b-7845-48de-8c12-61a7c1126976","Type":"ContainerStarted","Data":"692a1c69e65f3e76ed046e3d0d7fe1b7308d18e37f551acb5b6de41d8775fb95"}
Oct 30 20:25:19.295269 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:19.295215    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj" event={"ID":"b32f150b-7845-48de-8c12-61a7c1126976","Type":"ContainerStarted","Data":"4ca6d0351c8e6ac9c1f881fb55c12353faa95298ca134f4016e161ba4ded8bb7"}
Oct 30 20:25:19.417570 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:19.417519    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj" podStartSLOduration=1.417505476 podStartE2EDuration="1.417505476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:19.415512008 +0000 UTC m=+2558.147727398" watchObservedRunningTime="2026-10-30 20:25:19.417505476 +0000 UTC m=+2558.149720866"
Oct 30 20:25:19.460600 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:19.460561    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:25:28.353995 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:28.353954    5958 generic.go:356] "Generic (PLEG): container finished" podID="4b182630-ffc1-4770-a86c-3cdd5b763b92" containerID="fedb85bd1e7a0e8b19c9937cf3e9dc15567d32e16b9bac3ea89b765d2a36d329" exitCode=0
Oct 30 20:25:28.354511 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:28.354152    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2" event={"ID":"4b182630-ffc1-4770-a86c-3cdd5b763b92","Type":"ContainerDied","Data":"fedb85bd1e7a0e8b19c9937cf3e9dc15567d32e16b9bac3ea89b765d2a36d329"}
Oct 30 20:25:28.354718 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:28.354696    5958 scope.go:122] "RemoveContainer" containerID="fedb85bd1e7a0e8b19c9937cf3e9dc15567d32e16b9bac3ea89b765d2a36d329"
Oct 30 20:25:28.357150 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:28.357115    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:25:28.357288 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:28.357190    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:25:28.357336 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:28.357321    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:25:28.454282 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:28.454251    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:25:28.467363 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:28.467322    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:25:29.367692 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:29.367650    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2" event={"ID":"4b182630-ffc1-4770-a86c-3cdd5b763b92","Type":"ContainerStarted","Data":"7e238dcda9d4ecb62f1cb474939f2cee3f9f97ce08e435022449d3e7ebcd762f"}
Oct 30 20:25:29.379174 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:29.379147    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:25:29.859593 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:29.859546    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:25:29.888772 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:29.888743    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"]
Oct 30 20:25:29.889349 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:29.889336    5958 cpu_manager.go:414] "containerMap: removing container" podUID="590c5cad-401a-49be-a4b1-79892dc39d5d" containerName="istio-proxy"
Oct 30 20:25:29.889399 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:29.889351    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="590c5cad-401a-49be-a4b1-79892dc39d5d" containerName="istio-proxy"
Oct 30 20:25:29.893014 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:29.892995    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:25:29.962116 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:29.962087    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-453277\"/\"test-model-453277-kserve-self-signed-certs\""
Oct 30 20:25:29.962440 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:29.962422    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-453277\"/\"kube-root-ca.crt\""
Oct 30 20:25:29.962784 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:29.962727    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-453277\"/\"default-dockercfg-7hbpb\""
Oct 30 20:25:29.962933 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:29.962917    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-453277\"/\"openshift-service-ca.crt\""
Oct 30 20:25:29.966246 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:29.966220    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"]
Oct 30 20:25:29.998624 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:29.998595    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-home\") pod \"test-model-453277-kserve-df5f698dc-t5jfw\" (UID: \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\") " pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:25:29.998988 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:29.998966    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e07ac50-c692-49c7-a2c3-dcbc364693ae-tls-certs\") pod \"test-model-453277-kserve-df5f698dc-t5jfw\" (UID: \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\") " pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:25:29.999149 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:29.999138    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-dshm\") pod \"test-model-453277-kserve-df5f698dc-t5jfw\" (UID: \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\") " pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:25:29.999212 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:29.999204    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-tmp-dir\") pod \"test-model-453277-kserve-df5f698dc-t5jfw\" (UID: \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\") " pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:25:29.999293 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:29.999284    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4lj6\" (UniqueName: \"kubernetes.io/projected/5e07ac50-c692-49c7-a2c3-dcbc364693ae-kube-api-access-h4lj6\") pod \"test-model-453277-kserve-df5f698dc-t5jfw\" (UID: \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\") " pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:25:29.999409 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:29.999399    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-model-cache\") pod \"test-model-453277-kserve-df5f698dc-t5jfw\" (UID: \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\") " pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:25:30.100352 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:30.100319    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e07ac50-c692-49c7-a2c3-dcbc364693ae-tls-certs\") pod \"test-model-453277-kserve-df5f698dc-t5jfw\" (UID: \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\") " pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:25:30.100753 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:30.100716    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-dshm\") pod \"test-model-453277-kserve-df5f698dc-t5jfw\" (UID: \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\") " pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:25:30.100855 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:30.100764    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-tmp-dir\") pod \"test-model-453277-kserve-df5f698dc-t5jfw\" (UID: \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\") " pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:25:30.100855 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:30.100810    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4lj6\" (UniqueName: \"kubernetes.io/projected/5e07ac50-c692-49c7-a2c3-dcbc364693ae-kube-api-access-h4lj6\") pod \"test-model-453277-kserve-df5f698dc-t5jfw\" (UID: \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\") " pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:25:30.100943 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:30.100863    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-model-cache\") pod \"test-model-453277-kserve-df5f698dc-t5jfw\" (UID: \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\") " pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:25:30.101016 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:30.100999    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-home\") pod \"test-model-453277-kserve-df5f698dc-t5jfw\" (UID: \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\") " pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:25:30.101264 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:30.101241    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-tmp-dir\") pod \"test-model-453277-kserve-df5f698dc-t5jfw\" (UID: \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\") " pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:25:30.101412 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:30.101394    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-home\") pod \"test-model-453277-kserve-df5f698dc-t5jfw\" (UID: \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\") " pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:25:30.101540 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:30.101290    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-model-cache\") pod \"test-model-453277-kserve-df5f698dc-t5jfw\" (UID: \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\") " pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:25:30.103079 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:30.103055    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e07ac50-c692-49c7-a2c3-dcbc364693ae-tls-certs\") pod \"test-model-453277-kserve-df5f698dc-t5jfw\" (UID: \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\") " pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:25:30.103268 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:30.103249    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-dshm\") pod \"test-model-453277-kserve-df5f698dc-t5jfw\" (UID: \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\") " pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:25:30.163475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:30.163373    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4lj6\" (UniqueName: \"kubernetes.io/projected/5e07ac50-c692-49c7-a2c3-dcbc364693ae-kube-api-access-h4lj6\") pod \"test-model-453277-kserve-df5f698dc-t5jfw\" (UID: \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\") " pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:25:30.208014 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:30.207974    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:25:30.326577 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:30.320015    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:25:30.326577 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:30.322379    5958 patch_prober.go:29] interesting pod/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:25:30.326577 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:30.322427    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2" podUID="4b182630-ffc1-4770-a86c-3cdd5b763b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 20:25:30.612658 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:30.612393    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"]
Oct 30 20:25:30.614483 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:25:30.614436    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e07ac50_c692_49c7_a2c3_dcbc364693ae.slice/crio-7085986f97c0c57b9cdbda33a55679c0555530a1367b911d0e4b4b3c4b4b83fc WatchSource:0}: Error finding container 7085986f97c0c57b9cdbda33a55679c0555530a1367b911d0e4b4b3c4b4b83fc: Status 404 returned error can't find the container with id 7085986f97c0c57b9cdbda33a55679c0555530a1367b911d0e4b4b3c4b4b83fc
Oct 30 20:25:31.300336 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:31.300303    5958 patch_prober.go:29] interesting pod/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:25:31.300554 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:31.300364    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2" podUID="4b182630-ffc1-4770-a86c-3cdd5b763b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 20:25:31.383786 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:31.383748    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw" event={"ID":"5e07ac50-c692-49c7-a2c3-dcbc364693ae","Type":"ContainerStarted","Data":"fcc9816739a27284a13c6eb3ae13350efb453913ed806458b5d123e5ed3fbc6d"}
Oct 30 20:25:31.383944 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:31.383797    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw" event={"ID":"5e07ac50-c692-49c7-a2c3-dcbc364693ae","Type":"ContainerStarted","Data":"7085986f97c0c57b9cdbda33a55679c0555530a1367b911d0e4b4b3c4b4b83fc"}
Oct 30 20:25:31.525407 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:31.525348    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw" podStartSLOduration=2.5253316139999997 podStartE2EDuration="2.525331614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:31.523140189 +0000 UTC m=+2570.255355594" watchObservedRunningTime="2026-10-30 20:25:31.525331614 +0000 UTC m=+2570.257547006"
Oct 30 20:25:32.208635 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:32.208560    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:25:32.300316 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:32.300282    5958 patch_prober.go:29] interesting pod/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:25:32.300533 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:32.300339    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2" podUID="4b182630-ffc1-4770-a86c-3cdd5b763b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 20:25:33.300498 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:33.300423    5958 patch_prober.go:29] interesting pod/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 30 20:25:33.300885 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:33.300515    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2" podUID="4b182630-ffc1-4770-a86c-3cdd5b763b92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 30 20:25:34.385711 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:34.385674    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:25:34.387025 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:34.387003    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:25:36.607272 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:25:36.607236    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:25:40.208844 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:40.208805    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:25:40.221797 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:40.221769    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:25:40.458787 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:25:40.458755    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:26:04.079412 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:04.079378    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5623f05-snp8l"]
Oct 30 20:26:04.081795 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:04.081771    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5623f05-snp8l"]
Oct 30 20:26:05.399182 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:05.399143    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faafca8a-42dc-4613-89fe-c135d4a7acab" path="/var/lib/kubelet/pods/faafca8a-42dc-4613-89fe-c135d4a7acab/volumes"
Oct 30 20:26:06.605613 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:26:06.605576    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:26:09.980606 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:09.980575    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"]
Oct 30 20:26:25.001664 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.001574    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq" podUID="52f0b2bb-c5b0-4603-a096-09247fc67aa3" containerName="main" containerID="cri-o://d442dd6b142cd70efe3b263b42b973185ace90880888e9286d47d19a3b07663e" gracePeriod=45
Oct 30 20:26:25.265234 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.265166    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:26:25.360054 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.360018    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52f0b2bb-c5b0-4603-a096-09247fc67aa3-tls-certs\" (UniqueName: \"kubernetes.io/secret/52f0b2bb-c5b0-4603-a096-09247fc67aa3-tls-certs\") pod \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\" (UID: \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\") "
Oct 30 20:26:25.360255 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.360068    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-model-cache\") pod \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\" (UID: \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\") "
Oct 30 20:26:25.360255 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.360176    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52f0b2bb-c5b0-4603-a096-09247fc67aa3-kube-api-access-mscm8\" (UniqueName: \"kubernetes.io/projected/52f0b2bb-c5b0-4603-a096-09247fc67aa3-kube-api-access-mscm8\") pod \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\" (UID: \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\") "
Oct 30 20:26:25.360255 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.360238    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-home\" (UniqueName: \"kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-home\") pod \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\" (UID: \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\") "
Oct 30 20:26:25.360440 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.360266    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-dshm\") pod \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\" (UID: \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\") "
Oct 30 20:26:25.360440 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.360330    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-tmp-dir\") pod \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\" (UID: \"52f0b2bb-c5b0-4603-a096-09247fc67aa3\") "
Oct 30 20:26:25.360440 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.360373    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-model-cache" pod "52f0b2bb-c5b0-4603-a096-09247fc67aa3" (UID: "52f0b2bb-c5b0-4603-a096-09247fc67aa3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:25.360699 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.360522    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-home" pod "52f0b2bb-c5b0-4603-a096-09247fc67aa3" (UID: "52f0b2bb-c5b0-4603-a096-09247fc67aa3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:25.360699 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.360641    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-tmp-dir" pod "52f0b2bb-c5b0-4603-a096-09247fc67aa3" (UID: "52f0b2bb-c5b0-4603-a096-09247fc67aa3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:25.360822 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.360805    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-home\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:25.360887 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.360823    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-tmp-dir\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:25.360887 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.360833    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-model-cache\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:25.362631 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.362600    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52f0b2bb-c5b0-4603-a096-09247fc67aa3-tls-certs" pod "52f0b2bb-c5b0-4603-a096-09247fc67aa3" (UID: "52f0b2bb-c5b0-4603-a096-09247fc67aa3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:26:25.362754 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.362639    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-dshm" pod "52f0b2bb-c5b0-4603-a096-09247fc67aa3" (UID: "52f0b2bb-c5b0-4603-a096-09247fc67aa3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:25.362754 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.362648    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f0b2bb-c5b0-4603-a096-09247fc67aa3-kube-api-access-mscm8" pod "52f0b2bb-c5b0-4603-a096-09247fc67aa3" (UID: "52f0b2bb-c5b0-4603-a096-09247fc67aa3"). InnerVolumeSpecName "kube-api-access-mscm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:25.462864 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.462833    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f0b2bb-c5b0-4603-a096-09247fc67aa3-tls-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:25.463050 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.462939    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mscm8\" (UniqueName: \"kubernetes.io/projected/52f0b2bb-c5b0-4603-a096-09247fc67aa3-kube-api-access-mscm8\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:25.463050 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.462967    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f0b2bb-c5b0-4603-a096-09247fc67aa3-dshm\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:25.749311 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.749276    5958 generic.go:356] "Generic (PLEG): container finished" podID="52f0b2bb-c5b0-4603-a096-09247fc67aa3" containerID="d442dd6b142cd70efe3b263b42b973185ace90880888e9286d47d19a3b07663e" exitCode=0
Oct 30 20:26:25.749565 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.749319    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"
Oct 30 20:26:25.749565 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.749351    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq" event={"ID":"52f0b2bb-c5b0-4603-a096-09247fc67aa3","Type":"ContainerDied","Data":"d442dd6b142cd70efe3b263b42b973185ace90880888e9286d47d19a3b07663e"}
Oct 30 20:26:25.749565 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.749393    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq" event={"ID":"52f0b2bb-c5b0-4603-a096-09247fc67aa3","Type":"ContainerDied","Data":"ccc932ef62218789908201f5f2e8ff20aac7de50a820f6d9589fad6cf13cdd41"}
Oct 30 20:26:25.749565 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.749413    5958 scope.go:122] "RemoveContainer" containerID="d442dd6b142cd70efe3b263b42b973185ace90880888e9286d47d19a3b07663e"
Oct 30 20:26:25.759534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.759512    5958 scope.go:122] "RemoveContainer" containerID="d442dd6b142cd70efe3b263b42b973185ace90880888e9286d47d19a3b07663e"
Oct 30 20:26:25.759845 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:26:25.759824    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d442dd6b142cd70efe3b263b42b973185ace90880888e9286d47d19a3b07663e\": container with ID starting with d442dd6b142cd70efe3b263b42b973185ace90880888e9286d47d19a3b07663e not found: ID does not exist" containerID="d442dd6b142cd70efe3b263b42b973185ace90880888e9286d47d19a3b07663e"
Oct 30 20:26:25.759893 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.759854    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d442dd6b142cd70efe3b263b42b973185ace90880888e9286d47d19a3b07663e"} err="failed to get container status \"d442dd6b142cd70efe3b263b42b973185ace90880888e9286d47d19a3b07663e\": rpc error: code = NotFound desc = could not find container \"d442dd6b142cd70efe3b263b42b973185ace90880888e9286d47d19a3b07663e\": container with ID starting with d442dd6b142cd70efe3b263b42b973185ace90880888e9286d47d19a3b07663e not found: ID does not exist"
Oct 30 20:26:25.919966 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.919928    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"]
Oct 30 20:26:25.969872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:25.969834    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0051fb/test-model-0051fb-kserve-75f97765b7-gxcxq"]
Oct 30 20:26:27.401922 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:27.401881    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52f0b2bb-c5b0-4603-a096-09247fc67aa3" path="/var/lib/kubelet/pods/52f0b2bb-c5b0-4603-a096-09247fc67aa3/volumes"
Oct 30 20:26:36.608087 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:26:36.608049    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:26:41.427097 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:41.427050    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"]
Oct 30 20:26:41.427633 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:41.427297    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb" podUID="e6fa742a-7bee-4a79-b5a1-aa33c24552f6" containerName="istio-proxy" containerID="cri-o://eceece4314ddfd5c3a2b586fbb7ca992cebbcdebb601165e55dd0d936d024fa0" gracePeriod=30
Oct 30 20:26:41.766281 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:41.766197    5958 patch_prober.go:29] interesting pod/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:26:41.766281 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:41.766261    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb" podUID="e6fa742a-7bee-4a79-b5a1-aa33c24552f6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:26:44.251500 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:44.251443    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"]
Oct 30 20:26:44.251924 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:44.251661    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt" podUID="3bc76c29-f2ac-42be-8f69-7f3ac0f5869f" containerName="istio-proxy" containerID="cri-o://651f422b69acf6b753f6d4e983d6413b128ff72ed82b95460743d7fea4af45db" gracePeriod=30
Oct 30 20:26:46.176946 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.176903    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"]
Oct 30 20:26:46.693905 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.693873    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:26:46.833098 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.833065    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-podinfo\") pod \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") "
Oct 30 20:26:46.833307 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.833109    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-envoy\") pod \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") "
Oct 30 20:26:46.833307 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.833156    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-data\") pod \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") "
Oct 30 20:26:46.833307 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.833205    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istiod-ca-cert\") pod \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") "
Oct 30 20:26:46.833531 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.833340    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-kube-api-access-st82h\" (UniqueName: \"kubernetes.io/projected/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-kube-api-access-st82h\") pod \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") "
Oct 30 20:26:46.833531 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.833375    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-workload-certs\") pod \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") "
Oct 30 20:26:46.833531 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.833414    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-credential-socket\") pod \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") "
Oct 30 20:26:46.833531 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.833503    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-token\" (UniqueName: \"kubernetes.io/projected/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-token\") pod \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") "
Oct 30 20:26:46.833750 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.833565    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-workload-socket\") pod \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\" (UID: \"e6fa742a-7bee-4a79-b5a1-aa33c24552f6\") "
Oct 30 20:26:46.833750 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.833706    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-workload-certs" pod "e6fa742a-7bee-4a79-b5a1-aa33c24552f6" (UID: "e6fa742a-7bee-4a79-b5a1-aa33c24552f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:46.833750 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.833723    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-credential-socket" pod "e6fa742a-7bee-4a79-b5a1-aa33c24552f6" (UID: "e6fa742a-7bee-4a79-b5a1-aa33c24552f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:46.833911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.833757    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istiod-ca-cert" pod "e6fa742a-7bee-4a79-b5a1-aa33c24552f6" (UID: "e6fa742a-7bee-4a79-b5a1-aa33c24552f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:26:46.834003 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.833798    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-data" pod "e6fa742a-7bee-4a79-b5a1-aa33c24552f6" (UID: "e6fa742a-7bee-4a79-b5a1-aa33c24552f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:46.834083 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.834025    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-workload-socket" pod "e6fa742a-7bee-4a79-b5a1-aa33c24552f6" (UID: "e6fa742a-7bee-4a79-b5a1-aa33c24552f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:46.834346 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.834320    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:46.834514 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.834350    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:46.834514 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.834366    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:46.834514 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.834381    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:46.834514 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.834396    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:46.835673 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.835648    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-envoy" pod "e6fa742a-7bee-4a79-b5a1-aa33c24552f6" (UID: "e6fa742a-7bee-4a79-b5a1-aa33c24552f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:46.835756 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.835690    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-podinfo" pod "e6fa742a-7bee-4a79-b5a1-aa33c24552f6" (UID: "e6fa742a-7bee-4a79-b5a1-aa33c24552f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:26:46.835756 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.835741    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-kube-api-access-st82h" pod "e6fa742a-7bee-4a79-b5a1-aa33c24552f6" (UID: "e6fa742a-7bee-4a79-b5a1-aa33c24552f6"). InnerVolumeSpecName "kube-api-access-st82h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:46.835930 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.835906    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-token" pod "e6fa742a-7bee-4a79-b5a1-aa33c24552f6" (UID: "e6fa742a-7bee-4a79-b5a1-aa33c24552f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:46.894002 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.893960    5958 generic.go:356] "Generic (PLEG): container finished" podID="e6fa742a-7bee-4a79-b5a1-aa33c24552f6" containerID="eceece4314ddfd5c3a2b586fbb7ca992cebbcdebb601165e55dd0d936d024fa0" exitCode=0
Oct 30 20:26:46.894207 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.894011    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"
Oct 30 20:26:46.894207 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.894045    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb" event={"ID":"e6fa742a-7bee-4a79-b5a1-aa33c24552f6","Type":"ContainerDied","Data":"eceece4314ddfd5c3a2b586fbb7ca992cebbcdebb601165e55dd0d936d024fa0"}
Oct 30 20:26:46.894207 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.894114    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb" event={"ID":"e6fa742a-7bee-4a79-b5a1-aa33c24552f6","Type":"ContainerDied","Data":"0474177e2009796899e0302f1d65c2bdb6ab593bc04717ef407a566c0186423f"}
Oct 30 20:26:46.894207 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.894143    5958 scope.go:122] "RemoveContainer" containerID="eceece4314ddfd5c3a2b586fbb7ca992cebbcdebb601165e55dd0d936d024fa0"
Oct 30 20:26:46.905112 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.905089    5958 scope.go:122] "RemoveContainer" containerID="eceece4314ddfd5c3a2b586fbb7ca992cebbcdebb601165e55dd0d936d024fa0"
Oct 30 20:26:46.905513 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:26:46.905488    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eceece4314ddfd5c3a2b586fbb7ca992cebbcdebb601165e55dd0d936d024fa0\": container with ID starting with eceece4314ddfd5c3a2b586fbb7ca992cebbcdebb601165e55dd0d936d024fa0 not found: ID does not exist" containerID="eceece4314ddfd5c3a2b586fbb7ca992cebbcdebb601165e55dd0d936d024fa0"
Oct 30 20:26:46.905573 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.905524    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eceece4314ddfd5c3a2b586fbb7ca992cebbcdebb601165e55dd0d936d024fa0"} err="failed to get container status \"eceece4314ddfd5c3a2b586fbb7ca992cebbcdebb601165e55dd0d936d024fa0\": rpc error: code = NotFound desc = could not find container \"eceece4314ddfd5c3a2b586fbb7ca992cebbcdebb601165e55dd0d936d024fa0\": container with ID starting with eceece4314ddfd5c3a2b586fbb7ca992cebbcdebb601165e55dd0d936d024fa0 not found: ID does not exist"
Oct 30 20:26:46.935479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.935425    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:46.935660 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.935499    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:46.935660 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.935518    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:46.935660 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:46.935533    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st82h\" (UniqueName: \"kubernetes.io/projected/e6fa742a-7bee-4a79-b5a1-aa33c24552f6-kube-api-access-st82h\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:47.068605 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:47.068566    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"]
Oct 30 20:26:47.077495 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:47.077433    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-779921-openshift-default-5d5f4f84cf-lmlxb"]
Oct 30 20:26:47.399131 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:47.399086    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6fa742a-7bee-4a79-b5a1-aa33c24552f6" path="/var/lib/kubelet/pods/e6fa742a-7bee-4a79-b5a1-aa33c24552f6/volumes"
Oct 30 20:26:49.510559 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.510531    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:26:49.668730 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.668684    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-credential-socket\") pod \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") "
Oct 30 20:26:49.668730 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.668734    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-kube-api-access-v8tjz\" (UniqueName: \"kubernetes.io/projected/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-kube-api-access-v8tjz\") pod \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") "
Oct 30 20:26:49.668995 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.668781    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istiod-ca-cert\") pod \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") "
Oct 30 20:26:49.668995 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.668829    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-envoy\") pod \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") "
Oct 30 20:26:49.668995 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.668963    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-data\") pod \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") "
Oct 30 20:26:49.669155 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.669011    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-podinfo\") pod \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") "
Oct 30 20:26:49.669155 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.669057    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-credential-socket" pod "3bc76c29-f2ac-42be-8f69-7f3ac0f5869f" (UID: "3bc76c29-f2ac-42be-8f69-7f3ac0f5869f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:49.669155 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.669090    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-workload-certs\") pod \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") "
Oct 30 20:26:49.669155 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.669143    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-workload-socket\") pod \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") "
Oct 30 20:26:49.669364 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.669202    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-token\" (UniqueName: \"kubernetes.io/projected/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-token\") pod \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\" (UID: \"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f\") "
Oct 30 20:26:49.669364 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.669344    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-workload-certs" pod "3bc76c29-f2ac-42be-8f69-7f3ac0f5869f" (UID: "3bc76c29-f2ac-42be-8f69-7f3ac0f5869f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:49.669504 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.669391    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istiod-ca-cert" pod "3bc76c29-f2ac-42be-8f69-7f3ac0f5869f" (UID: "3bc76c29-f2ac-42be-8f69-7f3ac0f5869f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:26:49.669577 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.669524    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-data" pod "3bc76c29-f2ac-42be-8f69-7f3ac0f5869f" (UID: "3bc76c29-f2ac-42be-8f69-7f3ac0f5869f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:49.669577 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.669536    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-workload-socket" pod "3bc76c29-f2ac-42be-8f69-7f3ac0f5869f" (UID: "3bc76c29-f2ac-42be-8f69-7f3ac0f5869f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:49.670105 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.670078    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:49.670188 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.670110    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:49.670188 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.670127    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:49.670188 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.670140    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:49.670188 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.670154    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:49.671246 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.671226    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-envoy" pod "3bc76c29-f2ac-42be-8f69-7f3ac0f5869f" (UID: "3bc76c29-f2ac-42be-8f69-7f3ac0f5869f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:26:49.671633 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.671611    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-token" pod "3bc76c29-f2ac-42be-8f69-7f3ac0f5869f" (UID: "3bc76c29-f2ac-42be-8f69-7f3ac0f5869f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:49.671790 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.671768    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-podinfo" pod "3bc76c29-f2ac-42be-8f69-7f3ac0f5869f" (UID: "3bc76c29-f2ac-42be-8f69-7f3ac0f5869f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:26:49.671912 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.671795    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-kube-api-access-v8tjz" pod "3bc76c29-f2ac-42be-8f69-7f3ac0f5869f" (UID: "3bc76c29-f2ac-42be-8f69-7f3ac0f5869f"). InnerVolumeSpecName "kube-api-access-v8tjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:49.771973 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.771923    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8tjz\" (UniqueName: \"kubernetes.io/projected/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-kube-api-access-v8tjz\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:49.771973 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.771977    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:49.772186 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.771993    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:49.772186 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.772006    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:26:49.927543 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.927417    5958 generic.go:356] "Generic (PLEG): container finished" podID="3bc76c29-f2ac-42be-8f69-7f3ac0f5869f" containerID="651f422b69acf6b753f6d4e983d6413b128ff72ed82b95460743d7fea4af45db" exitCode=0
Oct 30 20:26:49.927543 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.927488    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"
Oct 30 20:26:49.927781 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.927483    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt" event={"ID":"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f","Type":"ContainerDied","Data":"651f422b69acf6b753f6d4e983d6413b128ff72ed82b95460743d7fea4af45db"}
Oct 30 20:26:49.927781 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.927641    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt" event={"ID":"3bc76c29-f2ac-42be-8f69-7f3ac0f5869f","Type":"ContainerDied","Data":"5eeff01c3e9dd92650b6789fb1b0c9fa7807949a0fbe0ff75aa71706c70e21a6"}
Oct 30 20:26:49.927781 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.927668    5958 scope.go:122] "RemoveContainer" containerID="651f422b69acf6b753f6d4e983d6413b128ff72ed82b95460743d7fea4af45db"
Oct 30 20:26:49.939207 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.939069    5958 scope.go:122] "RemoveContainer" containerID="651f422b69acf6b753f6d4e983d6413b128ff72ed82b95460743d7fea4af45db"
Oct 30 20:26:49.939534 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:26:49.939504    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"651f422b69acf6b753f6d4e983d6413b128ff72ed82b95460743d7fea4af45db\": container with ID starting with 651f422b69acf6b753f6d4e983d6413b128ff72ed82b95460743d7fea4af45db not found: ID does not exist" containerID="651f422b69acf6b753f6d4e983d6413b128ff72ed82b95460743d7fea4af45db"
Oct 30 20:26:49.939631 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:49.939544    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"651f422b69acf6b753f6d4e983d6413b128ff72ed82b95460743d7fea4af45db"} err="failed to get container status \"651f422b69acf6b753f6d4e983d6413b128ff72ed82b95460743d7fea4af45db\": rpc error: code = NotFound desc = could not find container \"651f422b69acf6b753f6d4e983d6413b128ff72ed82b95460743d7fea4af45db\": container with ID starting with 651f422b69acf6b753f6d4e983d6413b128ff72ed82b95460743d7fea4af45db not found: ID does not exist"
Oct 30 20:26:50.094843 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:50.094804    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"]
Oct 30 20:26:50.099165 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:50.099131    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z2qnt"]
Oct 30 20:26:51.399088 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:26:51.399054    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc76c29-f2ac-42be-8f69-7f3ac0f5869f" path="/var/lib/kubelet/pods/3bc76c29-f2ac-42be-8f69-7f3ac0f5869f/volumes"
Oct 30 20:27:01.198144 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:01.198086    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6" podUID="18ca8088-e3e8-4bd0-a302-2aab153ef504" containerName="main" containerID="cri-o://a9006c45f2fb51cefff2236cdbe66c3e03b547b7f0d52626e4f203bf24fd5d46" gracePeriod=45
Oct 30 20:27:01.493802 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:01.493773    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:27:01.522741 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:01.522700    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ca8088-e3e8-4bd0-a302-2aab153ef504-kube-api-access-slbnd\" (UniqueName: \"kubernetes.io/projected/18ca8088-e3e8-4bd0-a302-2aab153ef504-kube-api-access-slbnd\") pod \"18ca8088-e3e8-4bd0-a302-2aab153ef504\" (UID: \"18ca8088-e3e8-4bd0-a302-2aab153ef504\") "
Oct 30 20:27:01.522930 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:01.522761    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-dshm\" (UniqueName: \"kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-dshm\") pod \"18ca8088-e3e8-4bd0-a302-2aab153ef504\" (UID: \"18ca8088-e3e8-4bd0-a302-2aab153ef504\") "
Oct 30 20:27:01.522930 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:01.522792    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-model-cache\") pod \"18ca8088-e3e8-4bd0-a302-2aab153ef504\" (UID: \"18ca8088-e3e8-4bd0-a302-2aab153ef504\") "
Oct 30 20:27:01.522930 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:01.522847    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-home\" (UniqueName: \"kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-home\") pod \"18ca8088-e3e8-4bd0-a302-2aab153ef504\" (UID: \"18ca8088-e3e8-4bd0-a302-2aab153ef504\") "
Oct 30 20:27:01.523155 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:01.523019    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-tmp-dir\") pod \"18ca8088-e3e8-4bd0-a302-2aab153ef504\" (UID: \"18ca8088-e3e8-4bd0-a302-2aab153ef504\") "
Oct 30 20:27:01.523155 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:01.523053    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18ca8088-e3e8-4bd0-a302-2aab153ef504-tls-certs\" (UniqueName: \"kubernetes.io/secret/18ca8088-e3e8-4bd0-a302-2aab153ef504-tls-certs\") pod \"18ca8088-e3e8-4bd0-a302-2aab153ef504\" (UID: \"18ca8088-e3e8-4bd0-a302-2aab153ef504\") "
Oct 30 20:27:01.523155 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:01.523094    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-model-cache" pod "18ca8088-e3e8-4bd0-a302-2aab153ef504" (UID: "18ca8088-e3e8-4bd0-a302-2aab153ef504"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:01.523155 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:01.523117    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-home" pod "18ca8088-e3e8-4bd0-a302-2aab153ef504" (UID: "18ca8088-e3e8-4bd0-a302-2aab153ef504"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:01.523361 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:01.523274    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-tmp-dir" pod "18ca8088-e3e8-4bd0-a302-2aab153ef504" (UID: "18ca8088-e3e8-4bd0-a302-2aab153ef504"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:01.523699 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:01.523602    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-tmp-dir\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:01.523699 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:01.523628    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-model-cache\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:01.523699 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:01.523639    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-home\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:01.525205 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:01.525180    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ca8088-e3e8-4bd0-a302-2aab153ef504-kube-api-access-slbnd" pod "18ca8088-e3e8-4bd0-a302-2aab153ef504" (UID: "18ca8088-e3e8-4bd0-a302-2aab153ef504"). InnerVolumeSpecName "kube-api-access-slbnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:01.525315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:01.525265    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-dshm" pod "18ca8088-e3e8-4bd0-a302-2aab153ef504" (UID: "18ca8088-e3e8-4bd0-a302-2aab153ef504"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:27:01.525364 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:01.525340    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ca8088-e3e8-4bd0-a302-2aab153ef504-tls-certs" pod "18ca8088-e3e8-4bd0-a302-2aab153ef504" (UID: "18ca8088-e3e8-4bd0-a302-2aab153ef504"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:27:01.624842 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:01.624803    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18ca8088-e3e8-4bd0-a302-2aab153ef504-tls-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:01.624842 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:01.624845    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slbnd\" (UniqueName: \"kubernetes.io/projected/18ca8088-e3e8-4bd0-a302-2aab153ef504-kube-api-access-slbnd\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:01.625045 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:01.624861    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18ca8088-e3e8-4bd0-a302-2aab153ef504-dshm\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:02.007799 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:02.007766    5958 generic.go:356] "Generic (PLEG): container finished" podID="18ca8088-e3e8-4bd0-a302-2aab153ef504" containerID="a9006c45f2fb51cefff2236cdbe66c3e03b547b7f0d52626e4f203bf24fd5d46" exitCode=0
Oct 30 20:27:02.007990 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:02.007812    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"
Oct 30 20:27:02.007990 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:02.007848    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6" event={"ID":"18ca8088-e3e8-4bd0-a302-2aab153ef504","Type":"ContainerDied","Data":"a9006c45f2fb51cefff2236cdbe66c3e03b547b7f0d52626e4f203bf24fd5d46"}
Oct 30 20:27:02.007990 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:02.007896    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6" event={"ID":"18ca8088-e3e8-4bd0-a302-2aab153ef504","Type":"ContainerDied","Data":"d0a3a8a744266463f484c0f32552861b7cccbd4b27979eed5a27aff8342cb75e"}
Oct 30 20:27:02.007990 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:02.007918    5958 scope.go:122] "RemoveContainer" containerID="a9006c45f2fb51cefff2236cdbe66c3e03b547b7f0d52626e4f203bf24fd5d46"
Oct 30 20:27:02.024689 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:02.024263    5958 scope.go:122] "RemoveContainer" containerID="a9006c45f2fb51cefff2236cdbe66c3e03b547b7f0d52626e4f203bf24fd5d46"
Oct 30 20:27:02.026817 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:27:02.025060    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9006c45f2fb51cefff2236cdbe66c3e03b547b7f0d52626e4f203bf24fd5d46\": container with ID starting with a9006c45f2fb51cefff2236cdbe66c3e03b547b7f0d52626e4f203bf24fd5d46 not found: ID does not exist" containerID="a9006c45f2fb51cefff2236cdbe66c3e03b547b7f0d52626e4f203bf24fd5d46"
Oct 30 20:27:02.026817 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:02.025099    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9006c45f2fb51cefff2236cdbe66c3e03b547b7f0d52626e4f203bf24fd5d46"} err="failed to get container status \"a9006c45f2fb51cefff2236cdbe66c3e03b547b7f0d52626e4f203bf24fd5d46\": rpc error: code = NotFound desc = could not find container \"a9006c45f2fb51cefff2236cdbe66c3e03b547b7f0d52626e4f203bf24fd5d46\": container with ID starting with a9006c45f2fb51cefff2236cdbe66c3e03b547b7f0d52626e4f203bf24fd5d46 not found: ID does not exist"
Oct 30 20:27:02.175590 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:02.175559    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"]
Oct 30 20:27:02.182610 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:02.182577    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a2386c/test-model-a2386c-kserve-67cd4dfc85-4v7q6"]
Oct 30 20:27:03.399576 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:03.399541    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18ca8088-e3e8-4bd0-a302-2aab153ef504" path="/var/lib/kubelet/pods/18ca8088-e3e8-4bd0-a302-2aab153ef504/volumes"
Oct 30 20:27:06.607313 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:27:06.607274    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:27:11.061690 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:11.061649    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-575b12ca-mlv88"]
Oct 30 20:27:11.111951 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:11.111906    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-575b12ca-mlv88"]
Oct 30 20:27:11.399484 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:11.399406    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30153d8d-0b5e-4d18-a442-c82144050a85" path="/var/lib/kubelet/pods/30153d8d-0b5e-4d18-a442-c82144050a85/volumes"
Oct 30 20:27:16.332516 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.332477    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4be744cb-vh4h7"]
Oct 30 20:27:16.333089 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.333074    5958 cpu_manager.go:414] "containerMap: removing container" podUID="52f0b2bb-c5b0-4603-a096-09247fc67aa3" containerName="main"
Oct 30 20:27:16.333135 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.333091    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52f0b2bb-c5b0-4603-a096-09247fc67aa3" containerName="main"
Oct 30 20:27:16.333135 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.333114    5958 cpu_manager.go:414] "containerMap: removing container" podUID="e6fa742a-7bee-4a79-b5a1-aa33c24552f6" containerName="istio-proxy"
Oct 30 20:27:16.333135 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.333119    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6fa742a-7bee-4a79-b5a1-aa33c24552f6" containerName="istio-proxy"
Oct 30 20:27:16.333231 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.333135    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3bc76c29-f2ac-42be-8f69-7f3ac0f5869f" containerName="istio-proxy"
Oct 30 20:27:16.333231 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.333142    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc76c29-f2ac-42be-8f69-7f3ac0f5869f" containerName="istio-proxy"
Oct 30 20:27:16.333231 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.333161    5958 cpu_manager.go:414] "containerMap: removing container" podUID="18ca8088-e3e8-4bd0-a302-2aab153ef504" containerName="main"
Oct 30 20:27:16.333231 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.333166    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ca8088-e3e8-4bd0-a302-2aab153ef504" containerName="main"
Oct 30 20:27:16.333359 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.333243    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ca8088-e3e8-4bd0-a302-2aab153ef504" containerName="main"
Oct 30 20:27:16.333359 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.333254    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="52f0b2bb-c5b0-4603-a096-09247fc67aa3" containerName="main"
Oct 30 20:27:16.333359 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.333263    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6fa742a-7bee-4a79-b5a1-aa33c24552f6" containerName="istio-proxy"
Oct 30 20:27:16.333359 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.333272    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc76c29-f2ac-42be-8f69-7f3ac0f5869f" containerName="istio-proxy"
Oct 30 20:27:16.336938 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.336922    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be744cb-vh4h7"
Oct 30 20:27:16.394148 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.394111    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4be744cb-vh4h7"]
Oct 30 20:27:16.400906 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.400873    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b26180f-b498-4cc0-97e0-4b906a22f2f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4be744cb-vh4h7\" (UID: \"9b26180f-b498-4cc0-97e0-4b906a22f2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be744cb-vh4h7"
Oct 30 20:27:16.401063 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.400939    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lkh5\" (UniqueName: \"kubernetes.io/projected/9b26180f-b498-4cc0-97e0-4b906a22f2f6-kube-api-access-4lkh5\") pod \"maas-api-revoke-sub-4be744cb-vh4h7\" (UID: \"9b26180f-b498-4cc0-97e0-4b906a22f2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be744cb-vh4h7"
Oct 30 20:27:16.401137 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.401123    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b26180f-b498-4cc0-97e0-4b906a22f2f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4be744cb-vh4h7\" (UID: \"9b26180f-b498-4cc0-97e0-4b906a22f2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be744cb-vh4h7"
Oct 30 20:27:16.441069 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.441033    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:27:16.441069 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.441032    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:27:16.501832 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.501790    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b26180f-b498-4cc0-97e0-4b906a22f2f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4be744cb-vh4h7\" (UID: \"9b26180f-b498-4cc0-97e0-4b906a22f2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be744cb-vh4h7"
Oct 30 20:27:16.502030 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.501863    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b26180f-b498-4cc0-97e0-4b906a22f2f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4be744cb-vh4h7\" (UID: \"9b26180f-b498-4cc0-97e0-4b906a22f2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be744cb-vh4h7"
Oct 30 20:27:16.502030 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.501907    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lkh5\" (UniqueName: \"kubernetes.io/projected/9b26180f-b498-4cc0-97e0-4b906a22f2f6-kube-api-access-4lkh5\") pod \"maas-api-revoke-sub-4be744cb-vh4h7\" (UID: \"9b26180f-b498-4cc0-97e0-4b906a22f2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be744cb-vh4h7"
Oct 30 20:27:16.503552 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.502935    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b26180f-b498-4cc0-97e0-4b906a22f2f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4be744cb-vh4h7\" (UID: \"9b26180f-b498-4cc0-97e0-4b906a22f2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be744cb-vh4h7"
Oct 30 20:27:16.504787 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.504752    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b26180f-b498-4cc0-97e0-4b906a22f2f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4be744cb-vh4h7\" (UID: \"9b26180f-b498-4cc0-97e0-4b906a22f2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be744cb-vh4h7"
Oct 30 20:27:16.564214 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.564176    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lkh5\" (UniqueName: \"kubernetes.io/projected/9b26180f-b498-4cc0-97e0-4b906a22f2f6-kube-api-access-4lkh5\") pod \"maas-api-revoke-sub-4be744cb-vh4h7\" (UID: \"9b26180f-b498-4cc0-97e0-4b906a22f2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be744cb-vh4h7"
Oct 30 20:27:16.650721 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:16.650640    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be744cb-vh4h7"
Oct 30 20:27:17.027620 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:17.027591    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4be744cb-vh4h7"]
Oct 30 20:27:17.030124 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:27:17.030089    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b26180f_b498_4cc0_97e0_4b906a22f2f6.slice/crio-93dbf48fdcebea59a38609aacd6c1373d8ac4eb091115209bd787bd9c760a5f6 WatchSource:0}: Error finding container 93dbf48fdcebea59a38609aacd6c1373d8ac4eb091115209bd787bd9c760a5f6: Status 404 returned error can't find the container with id 93dbf48fdcebea59a38609aacd6c1373d8ac4eb091115209bd787bd9c760a5f6
Oct 30 20:27:17.107422 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:17.107380    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be744cb-vh4h7" event={"ID":"9b26180f-b498-4cc0-97e0-4b906a22f2f6","Type":"ContainerStarted","Data":"93dbf48fdcebea59a38609aacd6c1373d8ac4eb091115209bd787bd9c760a5f6"}
Oct 30 20:27:18.115728 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:18.115692    5958 generic.go:356] "Generic (PLEG): container finished" podID="9b26180f-b498-4cc0-97e0-4b906a22f2f6" containerID="dabdea84473f354f1449c19887d45c4cdbb497c4f5e8c69e7926169b77b93961" exitCode=0
Oct 30 20:27:18.116129 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:18.115731    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be744cb-vh4h7" event={"ID":"9b26180f-b498-4cc0-97e0-4b906a22f2f6","Type":"ContainerDied","Data":"dabdea84473f354f1449c19887d45c4cdbb497c4f5e8c69e7926169b77b93961"}
Oct 30 20:27:19.265360 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:19.265333    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be744cb-vh4h7"
Oct 30 20:27:19.334647 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:19.334612    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b26180f-b498-4cc0-97e0-4b906a22f2f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b26180f-b498-4cc0-97e0-4b906a22f2f6-maas-api-openshift-ca\") pod \"9b26180f-b498-4cc0-97e0-4b906a22f2f6\" (UID: \"9b26180f-b498-4cc0-97e0-4b906a22f2f6\") "
Oct 30 20:27:19.334821 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:19.334718    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b26180f-b498-4cc0-97e0-4b906a22f2f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b26180f-b498-4cc0-97e0-4b906a22f2f6-maas-api-xks-ca\") pod \"9b26180f-b498-4cc0-97e0-4b906a22f2f6\" (UID: \"9b26180f-b498-4cc0-97e0-4b906a22f2f6\") "
Oct 30 20:27:19.334821 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:19.334769    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b26180f-b498-4cc0-97e0-4b906a22f2f6-kube-api-access-4lkh5\" (UniqueName: \"kubernetes.io/projected/9b26180f-b498-4cc0-97e0-4b906a22f2f6-kube-api-access-4lkh5\") pod \"9b26180f-b498-4cc0-97e0-4b906a22f2f6\" (UID: \"9b26180f-b498-4cc0-97e0-4b906a22f2f6\") "
Oct 30 20:27:19.335045 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:19.335019    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b26180f-b498-4cc0-97e0-4b906a22f2f6-maas-api-openshift-ca" pod "9b26180f-b498-4cc0-97e0-4b906a22f2f6" (UID: "9b26180f-b498-4cc0-97e0-4b906a22f2f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:27:19.335177 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:19.335156    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b26180f-b498-4cc0-97e0-4b906a22f2f6-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:19.336935 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:19.336899    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b26180f-b498-4cc0-97e0-4b906a22f2f6-maas-api-xks-ca" pod "9b26180f-b498-4cc0-97e0-4b906a22f2f6" (UID: "9b26180f-b498-4cc0-97e0-4b906a22f2f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:27:19.337039 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:19.337001    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b26180f-b498-4cc0-97e0-4b906a22f2f6-kube-api-access-4lkh5" pod "9b26180f-b498-4cc0-97e0-4b906a22f2f6" (UID: "9b26180f-b498-4cc0-97e0-4b906a22f2f6"). InnerVolumeSpecName "kube-api-access-4lkh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:19.436477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:19.436429    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b26180f-b498-4cc0-97e0-4b906a22f2f6-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:19.436477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:19.436476    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lkh5\" (UniqueName: \"kubernetes.io/projected/9b26180f-b498-4cc0-97e0-4b906a22f2f6-kube-api-access-4lkh5\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:20.130289 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:20.130253    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be744cb-vh4h7" event={"ID":"9b26180f-b498-4cc0-97e0-4b906a22f2f6","Type":"ContainerDied","Data":"93dbf48fdcebea59a38609aacd6c1373d8ac4eb091115209bd787bd9c760a5f6"}
Oct 30 20:27:20.130486 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:20.130339    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93dbf48fdcebea59a38609aacd6c1373d8ac4eb091115209bd787bd9c760a5f6"
Oct 30 20:27:20.130486 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:20.130262    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4be744cb-vh4h7"
Oct 30 20:27:25.965540 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:25.965500    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8"]
Oct 30 20:27:30.966022 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:30.965940    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8" podUID="f57472cf-a3f7-486d-9368-1c692e31bedc" containerName="payload-processing" containerID="cri-o://c1ca32eecf12ef718a01082c8a61058a1d075de1dcec3fab494a87790fba2f30" gracePeriod=25
Oct 30 20:27:33.197330 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:33.197286    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8" podUID="f57472cf-a3f7-486d-9368-1c692e31bedc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:27:36.605855 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:27:36.605809    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:27:41.871674 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:41.871644    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 20:27:41.883639 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:41.883609    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 20:27:42.536892 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:42.536856    5958 scope.go:122] "RemoveContainer" containerID="cb1079cbfd7132da9b4c79351bde233b59a26ae23585c7fe9db9faf90199c5ea"
Oct 30 20:27:43.197282 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:43.197240    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8" podUID="f57472cf-a3f7-486d-9368-1c692e31bedc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:27:48.860805 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:48.860761    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"]
Oct 30 20:27:51.237170 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:51.237100    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8"
Oct 30 20:27:51.314704 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:51.314671    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f57472cf-a3f7-486d-9368-1c692e31bedc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57472cf-a3f7-486d-9368-1c692e31bedc-plugins-config-volume\") pod \"f57472cf-a3f7-486d-9368-1c692e31bedc\" (UID: \"f57472cf-a3f7-486d-9368-1c692e31bedc\") "
Oct 30 20:27:51.314864 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:51.314828    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f57472cf-a3f7-486d-9368-1c692e31bedc-kube-api-access-wfggd\" (UniqueName: \"kubernetes.io/projected/f57472cf-a3f7-486d-9368-1c692e31bedc-kube-api-access-wfggd\") pod \"f57472cf-a3f7-486d-9368-1c692e31bedc\" (UID: \"f57472cf-a3f7-486d-9368-1c692e31bedc\") "
Oct 30 20:27:51.315104 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:51.315075    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f57472cf-a3f7-486d-9368-1c692e31bedc-plugins-config-volume" pod "f57472cf-a3f7-486d-9368-1c692e31bedc" (UID: "f57472cf-a3f7-486d-9368-1c692e31bedc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:27:51.315334 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:51.315318    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57472cf-a3f7-486d-9368-1c692e31bedc-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:51.316957 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:51.316931    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f57472cf-a3f7-486d-9368-1c692e31bedc-kube-api-access-wfggd" pod "f57472cf-a3f7-486d-9368-1c692e31bedc" (UID: "f57472cf-a3f7-486d-9368-1c692e31bedc"). InnerVolumeSpecName "kube-api-access-wfggd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:27:51.339378 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:51.339340    5958 generic.go:356] "Generic (PLEG): container finished" podID="f57472cf-a3f7-486d-9368-1c692e31bedc" containerID="c1ca32eecf12ef718a01082c8a61058a1d075de1dcec3fab494a87790fba2f30" exitCode=0
Oct 30 20:27:51.339577 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:51.339394    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8"
Oct 30 20:27:51.339577 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:51.339439    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8" event={"ID":"f57472cf-a3f7-486d-9368-1c692e31bedc","Type":"ContainerDied","Data":"c1ca32eecf12ef718a01082c8a61058a1d075de1dcec3fab494a87790fba2f30"}
Oct 30 20:27:51.339577 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:51.339520    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8" event={"ID":"f57472cf-a3f7-486d-9368-1c692e31bedc","Type":"ContainerDied","Data":"19776372fe411cfde39e9300933bbe917e57cdd7a965a3f8b000b20ea2e765d1"}
Oct 30 20:27:51.339577 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:51.339546    5958 scope.go:122] "RemoveContainer" containerID="c1ca32eecf12ef718a01082c8a61058a1d075de1dcec3fab494a87790fba2f30"
Oct 30 20:27:51.351815 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:51.351792    5958 scope.go:122] "RemoveContainer" containerID="c1ca32eecf12ef718a01082c8a61058a1d075de1dcec3fab494a87790fba2f30"
Oct 30 20:27:51.352134 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:27:51.352115    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1ca32eecf12ef718a01082c8a61058a1d075de1dcec3fab494a87790fba2f30\": container with ID starting with c1ca32eecf12ef718a01082c8a61058a1d075de1dcec3fab494a87790fba2f30 not found: ID does not exist" containerID="c1ca32eecf12ef718a01082c8a61058a1d075de1dcec3fab494a87790fba2f30"
Oct 30 20:27:51.352184 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:51.352144    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ca32eecf12ef718a01082c8a61058a1d075de1dcec3fab494a87790fba2f30"} err="failed to get container status \"c1ca32eecf12ef718a01082c8a61058a1d075de1dcec3fab494a87790fba2f30\": rpc error: code = NotFound desc = could not find container \"c1ca32eecf12ef718a01082c8a61058a1d075de1dcec3fab494a87790fba2f30\": container with ID starting with c1ca32eecf12ef718a01082c8a61058a1d075de1dcec3fab494a87790fba2f30 not found: ID does not exist"
Oct 30 20:27:51.416745 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:51.416707    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfggd\" (UniqueName: \"kubernetes.io/projected/f57472cf-a3f7-486d-9368-1c692e31bedc-kube-api-access-wfggd\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:27:51.467057 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:51.467021    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8"]
Oct 30 20:27:51.470950 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:51.470916    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a2386c-5f97855cf6-6vtz8"]
Oct 30 20:27:53.410585 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:27:53.410541    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f57472cf-a3f7-486d-9368-1c692e31bedc" path="/var/lib/kubelet/pods/f57472cf-a3f7-486d-9368-1c692e31bedc/volumes"
Oct 30 20:28:01.475795 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:01.475749    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"]
Oct 30 20:28:01.476277 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:01.476028    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7" podUID="04b5051c-1bff-4550-b442-7f0b7622a271" containerName="istio-proxy" containerID="cri-o://2c09c75f16955668a6ec949c6edc350888d135419a38e7d621f19360b8c44a1a" gracePeriod=30
Oct 30 20:28:02.389400 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:02.389347    5958 patch_prober.go:29] interesting pod/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:28:02.389610 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:02.389407    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7" podUID="04b5051c-1bff-4550-b442-7f0b7622a271" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:28:03.883207 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:03.883160    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj" podUID="b32f150b-7845-48de-8c12-61a7c1126976" containerName="main" containerID="cri-o://692a1c69e65f3e76ed046e3d0d7fe1b7308d18e37f551acb5b6de41d8775fb95" gracePeriod=45
Oct 30 20:28:04.146743 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.146657    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:28:04.187797 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.187757    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-dshm\") pod \"b32f150b-7845-48de-8c12-61a7c1126976\" (UID: \"b32f150b-7845-48de-8c12-61a7c1126976\") "
Oct 30 20:28:04.187970 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.187825    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-home\" (UniqueName: \"kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-home\") pod \"b32f150b-7845-48de-8c12-61a7c1126976\" (UID: \"b32f150b-7845-48de-8c12-61a7c1126976\") "
Oct 30 20:28:04.187970 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.187917    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b32f150b-7845-48de-8c12-61a7c1126976-tls-certs\" (UniqueName: \"kubernetes.io/secret/b32f150b-7845-48de-8c12-61a7c1126976-tls-certs\") pod \"b32f150b-7845-48de-8c12-61a7c1126976\" (UID: \"b32f150b-7845-48de-8c12-61a7c1126976\") "
Oct 30 20:28:04.187970 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.187938    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-tmp-dir\") pod \"b32f150b-7845-48de-8c12-61a7c1126976\" (UID: \"b32f150b-7845-48de-8c12-61a7c1126976\") "
Oct 30 20:28:04.188127 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.188025    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b32f150b-7845-48de-8c12-61a7c1126976-kube-api-access-vx8hs\" (UniqueName: \"kubernetes.io/projected/b32f150b-7845-48de-8c12-61a7c1126976-kube-api-access-vx8hs\") pod \"b32f150b-7845-48de-8c12-61a7c1126976\" (UID: \"b32f150b-7845-48de-8c12-61a7c1126976\") "
Oct 30 20:28:04.188127 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.188058    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-model-cache\") pod \"b32f150b-7845-48de-8c12-61a7c1126976\" (UID: \"b32f150b-7845-48de-8c12-61a7c1126976\") "
Oct 30 20:28:04.188231 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.188125    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-home" pod "b32f150b-7845-48de-8c12-61a7c1126976" (UID: "b32f150b-7845-48de-8c12-61a7c1126976"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:04.188326 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.188293    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-tmp-dir" pod "b32f150b-7845-48de-8c12-61a7c1126976" (UID: "b32f150b-7845-48de-8c12-61a7c1126976"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:04.188514 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.188486    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-model-cache" pod "b32f150b-7845-48de-8c12-61a7c1126976" (UID: "b32f150b-7845-48de-8c12-61a7c1126976"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:04.188713 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.188603    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-tmp-dir\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:04.188836 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.188752    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-home\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:04.190332 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.190307    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-dshm" pod "b32f150b-7845-48de-8c12-61a7c1126976" (UID: "b32f150b-7845-48de-8c12-61a7c1126976"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:04.190429 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.190324    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b32f150b-7845-48de-8c12-61a7c1126976-tls-certs" pod "b32f150b-7845-48de-8c12-61a7c1126976" (UID: "b32f150b-7845-48de-8c12-61a7c1126976"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:28:04.190535 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.190427    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32f150b-7845-48de-8c12-61a7c1126976-kube-api-access-vx8hs" pod "b32f150b-7845-48de-8c12-61a7c1126976" (UID: "b32f150b-7845-48de-8c12-61a7c1126976"). InnerVolumeSpecName "kube-api-access-vx8hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:04.290340 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.290300    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-dshm\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:04.290340 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.290336    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b32f150b-7845-48de-8c12-61a7c1126976-tls-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:04.290587 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.290348    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx8hs\" (UniqueName: \"kubernetes.io/projected/b32f150b-7845-48de-8c12-61a7c1126976-kube-api-access-vx8hs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:04.290587 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.290359    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b32f150b-7845-48de-8c12-61a7c1126976-model-cache\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:04.447833 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.447735    5958 generic.go:356] "Generic (PLEG): container finished" podID="b32f150b-7845-48de-8c12-61a7c1126976" containerID="692a1c69e65f3e76ed046e3d0d7fe1b7308d18e37f551acb5b6de41d8775fb95" exitCode=0
Oct 30 20:28:04.447833 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.447784    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"
Oct 30 20:28:04.448061 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.447826    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj" event={"ID":"b32f150b-7845-48de-8c12-61a7c1126976","Type":"ContainerDied","Data":"692a1c69e65f3e76ed046e3d0d7fe1b7308d18e37f551acb5b6de41d8775fb95"}
Oct 30 20:28:04.448061 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.447878    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj" event={"ID":"b32f150b-7845-48de-8c12-61a7c1126976","Type":"ContainerDied","Data":"4ca6d0351c8e6ac9c1f881fb55c12353faa95298ca134f4016e161ba4ded8bb7"}
Oct 30 20:28:04.448061 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.447899    5958 scope.go:122] "RemoveContainer" containerID="692a1c69e65f3e76ed046e3d0d7fe1b7308d18e37f551acb5b6de41d8775fb95"
Oct 30 20:28:04.458204 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.458186    5958 scope.go:122] "RemoveContainer" containerID="692a1c69e65f3e76ed046e3d0d7fe1b7308d18e37f551acb5b6de41d8775fb95"
Oct 30 20:28:04.458433 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:28:04.458415    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"692a1c69e65f3e76ed046e3d0d7fe1b7308d18e37f551acb5b6de41d8775fb95\": container with ID starting with 692a1c69e65f3e76ed046e3d0d7fe1b7308d18e37f551acb5b6de41d8775fb95 not found: ID does not exist" containerID="692a1c69e65f3e76ed046e3d0d7fe1b7308d18e37f551acb5b6de41d8775fb95"
Oct 30 20:28:04.458545 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.458441    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"692a1c69e65f3e76ed046e3d0d7fe1b7308d18e37f551acb5b6de41d8775fb95"} err="failed to get container status \"692a1c69e65f3e76ed046e3d0d7fe1b7308d18e37f551acb5b6de41d8775fb95\": rpc error: code = NotFound desc = could not find container \"692a1c69e65f3e76ed046e3d0d7fe1b7308d18e37f551acb5b6de41d8775fb95\": container with ID starting with 692a1c69e65f3e76ed046e3d0d7fe1b7308d18e37f551acb5b6de41d8775fb95 not found: ID does not exist"
Oct 30 20:28:04.621337 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.621296    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"]
Oct 30 20:28:04.623791 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:04.623767    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-01e8dc/test-model-01e8dc-kserve-f8f46c89c-2ctnj"]
Oct 30 20:28:05.403262 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:05.403224    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b32f150b-7845-48de-8c12-61a7c1126976" path="/var/lib/kubelet/pods/b32f150b-7845-48de-8c12-61a7c1126976/volumes"
Oct 30 20:28:06.607047 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:28:06.607010    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:28:06.754464 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.754389    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:28:06.820502 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.820417    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04b5051c-1bff-4550-b442-7f0b7622a271-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04b5051c-1bff-4550-b442-7f0b7622a271-istio-podinfo\") pod \"04b5051c-1bff-4550-b442-7f0b7622a271\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") "
Oct 30 20:28:06.820502 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.820513    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-workload-socket\") pod \"04b5051c-1bff-4550-b442-7f0b7622a271\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") "
Oct 30 20:28:06.820771 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.820565    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04b5051c-1bff-4550-b442-7f0b7622a271-istio-token\" (UniqueName: \"kubernetes.io/projected/04b5051c-1bff-4550-b442-7f0b7622a271-istio-token\") pod \"04b5051c-1bff-4550-b442-7f0b7622a271\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") "
Oct 30 20:28:06.820771 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.820612    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04b5051c-1bff-4550-b442-7f0b7622a271-kube-api-access-xk6j6\" (UniqueName: \"kubernetes.io/projected/04b5051c-1bff-4550-b442-7f0b7622a271-kube-api-access-xk6j6\") pod \"04b5051c-1bff-4550-b442-7f0b7622a271\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") "
Oct 30 20:28:06.820771 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.820674    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-workload-certs\") pod \"04b5051c-1bff-4550-b442-7f0b7622a271\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") "
Oct 30 20:28:06.820771 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.820730    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-credential-socket\") pod \"04b5051c-1bff-4550-b442-7f0b7622a271\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") "
Oct 30 20:28:06.820974 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.820824    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-workload-socket" pod "04b5051c-1bff-4550-b442-7f0b7622a271" (UID: "04b5051c-1bff-4550-b442-7f0b7622a271"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:06.820974 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.820835    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-istio-data\") pod \"04b5051c-1bff-4550-b442-7f0b7622a271\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") "
Oct 30 20:28:06.820974 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.820934    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04b5051c-1bff-4550-b442-7f0b7622a271-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04b5051c-1bff-4550-b442-7f0b7622a271-istiod-ca-cert\") pod \"04b5051c-1bff-4550-b442-7f0b7622a271\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") "
Oct 30 20:28:06.820974 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.820969    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-istio-envoy\") pod \"04b5051c-1bff-4550-b442-7f0b7622a271\" (UID: \"04b5051c-1bff-4550-b442-7f0b7622a271\") "
Oct 30 20:28:06.821177 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.820995    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-workload-certs" pod "04b5051c-1bff-4550-b442-7f0b7622a271" (UID: "04b5051c-1bff-4550-b442-7f0b7622a271"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:06.821177 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.821021    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-credential-socket" pod "04b5051c-1bff-4550-b442-7f0b7622a271" (UID: "04b5051c-1bff-4550-b442-7f0b7622a271"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:06.821177 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.821074    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-istio-data" pod "04b5051c-1bff-4550-b442-7f0b7622a271" (UID: "04b5051c-1bff-4550-b442-7f0b7622a271"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:06.821335 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.821310    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04b5051c-1bff-4550-b442-7f0b7622a271-istiod-ca-cert" pod "04b5051c-1bff-4550-b442-7f0b7622a271" (UID: "04b5051c-1bff-4550-b442-7f0b7622a271"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:28:06.821729 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.821705    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:06.821801 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.821734    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:06.821801 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.821751    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:06.821801 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.821766    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:06.821801 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.821780    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04b5051c-1bff-4550-b442-7f0b7622a271-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:06.823110 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.823077    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b5051c-1bff-4550-b442-7f0b7622a271-kube-api-access-xk6j6" pod "04b5051c-1bff-4550-b442-7f0b7622a271" (UID: "04b5051c-1bff-4550-b442-7f0b7622a271"). InnerVolumeSpecName "kube-api-access-xk6j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:06.823231 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.823140    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-istio-envoy" pod "04b5051c-1bff-4550-b442-7f0b7622a271" (UID: "04b5051c-1bff-4550-b442-7f0b7622a271"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:06.823387 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.823356    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04b5051c-1bff-4550-b442-7f0b7622a271-istio-podinfo" pod "04b5051c-1bff-4550-b442-7f0b7622a271" (UID: "04b5051c-1bff-4550-b442-7f0b7622a271"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:28:06.823489 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.823413    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b5051c-1bff-4550-b442-7f0b7622a271-istio-token" pod "04b5051c-1bff-4550-b442-7f0b7622a271" (UID: "04b5051c-1bff-4550-b442-7f0b7622a271"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:06.888376 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.888344    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"]
Oct 30 20:28:06.909277 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.909246    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"]
Oct 30 20:28:06.923157 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.923121    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04b5051c-1bff-4550-b442-7f0b7622a271-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:06.923157 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.923150    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk6j6\" (UniqueName: \"kubernetes.io/projected/04b5051c-1bff-4550-b442-7f0b7622a271-kube-api-access-xk6j6\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:06.923157 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.923161    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04b5051c-1bff-4550-b442-7f0b7622a271-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:06.923386 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:06.923169    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04b5051c-1bff-4550-b442-7f0b7622a271-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:07.469271 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:07.469232    5958 generic.go:356] "Generic (PLEG): container finished" podID="04b5051c-1bff-4550-b442-7f0b7622a271" containerID="2c09c75f16955668a6ec949c6edc350888d135419a38e7d621f19360b8c44a1a" exitCode=0
Oct 30 20:28:07.469440 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:07.469290    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"
Oct 30 20:28:07.469440 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:07.469326    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7" event={"ID":"04b5051c-1bff-4550-b442-7f0b7622a271","Type":"ContainerDied","Data":"2c09c75f16955668a6ec949c6edc350888d135419a38e7d621f19360b8c44a1a"}
Oct 30 20:28:07.469440 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:07.469385    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7" event={"ID":"04b5051c-1bff-4550-b442-7f0b7622a271","Type":"ContainerDied","Data":"11b1cb1a72d75178197d1dd5c083d786688c0dc4d58aa556e3a3b296fe66bc53"}
Oct 30 20:28:07.469440 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:07.469408    5958 scope.go:122] "RemoveContainer" containerID="2c09c75f16955668a6ec949c6edc350888d135419a38e7d621f19360b8c44a1a"
Oct 30 20:28:07.479067 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:07.479041    5958 scope.go:122] "RemoveContainer" containerID="265b226f7c326898944e7502af9464b56f87ce3e1929059ef9f48424883a29b9"
Oct 30 20:28:07.487968 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:07.487939    5958 scope.go:122] "RemoveContainer" containerID="2c09c75f16955668a6ec949c6edc350888d135419a38e7d621f19360b8c44a1a"
Oct 30 20:28:07.488249 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:28:07.488229    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c09c75f16955668a6ec949c6edc350888d135419a38e7d621f19360b8c44a1a\": container with ID starting with 2c09c75f16955668a6ec949c6edc350888d135419a38e7d621f19360b8c44a1a not found: ID does not exist" containerID="2c09c75f16955668a6ec949c6edc350888d135419a38e7d621f19360b8c44a1a"
Oct 30 20:28:07.488344 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:07.488259    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c09c75f16955668a6ec949c6edc350888d135419a38e7d621f19360b8c44a1a"} err="failed to get container status \"2c09c75f16955668a6ec949c6edc350888d135419a38e7d621f19360b8c44a1a\": rpc error: code = NotFound desc = could not find container \"2c09c75f16955668a6ec949c6edc350888d135419a38e7d621f19360b8c44a1a\": container with ID starting with 2c09c75f16955668a6ec949c6edc350888d135419a38e7d621f19360b8c44a1a not found: ID does not exist"
Oct 30 20:28:07.488344 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:07.488284    5958 scope.go:122] "RemoveContainer" containerID="265b226f7c326898944e7502af9464b56f87ce3e1929059ef9f48424883a29b9"
Oct 30 20:28:07.488572 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:28:07.488551    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"265b226f7c326898944e7502af9464b56f87ce3e1929059ef9f48424883a29b9\": container with ID starting with 265b226f7c326898944e7502af9464b56f87ce3e1929059ef9f48424883a29b9 not found: ID does not exist" containerID="265b226f7c326898944e7502af9464b56f87ce3e1929059ef9f48424883a29b9"
Oct 30 20:28:07.488628 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:07.488577    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"265b226f7c326898944e7502af9464b56f87ce3e1929059ef9f48424883a29b9"} err="failed to get container status \"265b226f7c326898944e7502af9464b56f87ce3e1929059ef9f48424883a29b9\": rpc error: code = NotFound desc = could not find container \"265b226f7c326898944e7502af9464b56f87ce3e1929059ef9f48424883a29b9\": container with ID starting with 265b226f7c326898944e7502af9464b56f87ce3e1929059ef9f48424883a29b9 not found: ID does not exist"
Oct 30 20:28:07.635381 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:07.635332    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"]
Oct 30 20:28:07.685417 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:07.685322    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a2386c-openshift-default-cb748d48c-sspm7"]
Oct 30 20:28:09.398274 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:09.398235    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04b5051c-1bff-4550-b442-7f0b7622a271" path="/var/lib/kubelet/pods/04b5051c-1bff-4550-b442-7f0b7622a271/volumes"
Oct 30 20:28:13.926231 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:13.926188    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv"]
Oct 30 20:28:14.034009 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:14.033965    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns"]
Oct 30 20:28:18.927484 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:18.927429    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv" podUID="e0d98ef7-ed5f-4f06-b7f8-0ff074f41033" containerName="payload-processing" containerID="cri-o://b2f7158ac1b49f7e5189c0d4f2f97b742369b8db0533d029446313106c15f0b0" gracePeriod=25
Oct 30 20:28:19.035314 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:19.035243    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns" podUID="58884900-a56f-40e1-8ce3-aea388a61245" containerName="payload-pre-processing" containerID="cri-o://a19c64ec02ca84415ca31e7962bc489858e2308f2a30a6f21319ff19f1c3e0a5" gracePeriod=25
Oct 30 20:28:21.909478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:21.909413    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main" containerID="cri-o://8518e28d69355a3546b16bbd82191ffbc89666665a0ee210293d7f14123630ba" gracePeriod=45
Oct 30 20:28:21.929603 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:21.929570    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main" containerID="cri-o://8cc6ee3c0a5a70aab4aaeba2a4a86f5545e39cd420ff5909029fcd255e24dc94" gracePeriod=45
Oct 30 20:28:21.995832 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:21.995795    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": dial tcp 10.134.0.72:8000: connect: connection refused"
Oct 30 20:28:22.238013 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.237986    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:28:22.241934 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.241905    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 20:28:22.302366 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.302338    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-home\" (UniqueName: \"kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-home\") pod \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\" (UID: \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\") "
Oct 30 20:28:22.302562 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.302396    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-tmp-dir\") pod \"3fff08e6-97c4-4572-9aad-95ae489fbffb\" (UID: \"3fff08e6-97c4-4572-9aad-95ae489fbffb\") "
Oct 30 20:28:22.302562 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.302421    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-model-cache\") pod \"3fff08e6-97c4-4572-9aad-95ae489fbffb\" (UID: \"3fff08e6-97c4-4572-9aad-95ae489fbffb\") "
Oct 30 20:28:22.302562 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.302494    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-tls-certs\" (UniqueName: \"kubernetes.io/secret/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-tls-certs\") pod \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\" (UID: \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\") "
Oct 30 20:28:22.302562 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.302523    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-dshm\") pod \"3fff08e6-97c4-4572-9aad-95ae489fbffb\" (UID: \"3fff08e6-97c4-4572-9aad-95ae489fbffb\") "
Oct 30 20:28:22.302562 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.302555    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-kube-api-access-5rz9r\" (UniqueName: \"kubernetes.io/projected/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-kube-api-access-5rz9r\") pod \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\" (UID: \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\") "
Oct 30 20:28:22.302833 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.302615    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-dshm\") pod \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\" (UID: \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\") "
Oct 30 20:28:22.302833 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.302644    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-tmp-dir\") pod \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\" (UID: \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\") "
Oct 30 20:28:22.302833 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.302682    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-home" pod "4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" (UID: "4aee1f03-06f6-485b-ac44-f60b7ba4ca6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:22.302833 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.302702    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fff08e6-97c4-4572-9aad-95ae489fbffb-tls-certs\" (UniqueName: \"kubernetes.io/secret/3fff08e6-97c4-4572-9aad-95ae489fbffb-tls-certs\") pod \"3fff08e6-97c4-4572-9aad-95ae489fbffb\" (UID: \"3fff08e6-97c4-4572-9aad-95ae489fbffb\") "
Oct 30 20:28:22.302833 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.302791    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-home\" (UniqueName: \"kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-home\") pod \"3fff08e6-97c4-4572-9aad-95ae489fbffb\" (UID: \"3fff08e6-97c4-4572-9aad-95ae489fbffb\") "
Oct 30 20:28:22.302833 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.302801    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-model-cache" pod "3fff08e6-97c4-4572-9aad-95ae489fbffb" (UID: "3fff08e6-97c4-4572-9aad-95ae489fbffb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:22.303145 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.302867    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-model-cache\") pod \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\" (UID: \"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b\") "
Oct 30 20:28:22.303145 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.302902    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fff08e6-97c4-4572-9aad-95ae489fbffb-kube-api-access-rqktl\" (UniqueName: \"kubernetes.io/projected/3fff08e6-97c4-4572-9aad-95ae489fbffb-kube-api-access-rqktl\") pod \"3fff08e6-97c4-4572-9aad-95ae489fbffb\" (UID: \"3fff08e6-97c4-4572-9aad-95ae489fbffb\") "
Oct 30 20:28:22.303145 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.303027    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-home" pod "3fff08e6-97c4-4572-9aad-95ae489fbffb" (UID: "3fff08e6-97c4-4572-9aad-95ae489fbffb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:22.303285 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.303214    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-tmp-dir" pod "4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" (UID: "4aee1f03-06f6-485b-ac44-f60b7ba4ca6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:22.303340 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.303320    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-tmp-dir" pod "3fff08e6-97c4-4572-9aad-95ae489fbffb" (UID: "3fff08e6-97c4-4572-9aad-95ae489fbffb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:22.303645 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.303620    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-home\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:22.303747 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.303648    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-tmp-dir\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:22.303747 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.303661    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-model-cache\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:22.303747 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.303673    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-tmp-dir\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:22.303747 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.303686    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-home\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:22.303971 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.303744    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-model-cache" pod "4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" (UID: "4aee1f03-06f6-485b-ac44-f60b7ba4ca6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:22.306333 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.306308    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-tls-certs" pod "4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" (UID: "4aee1f03-06f6-485b-ac44-f60b7ba4ca6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:28:22.306860 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.306819    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fff08e6-97c4-4572-9aad-95ae489fbffb-tls-certs" pod "3fff08e6-97c4-4572-9aad-95ae489fbffb" (UID: "3fff08e6-97c4-4572-9aad-95ae489fbffb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:28:22.307195 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.307157    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fff08e6-97c4-4572-9aad-95ae489fbffb-kube-api-access-rqktl" pod "3fff08e6-97c4-4572-9aad-95ae489fbffb" (UID: "3fff08e6-97c4-4572-9aad-95ae489fbffb"). InnerVolumeSpecName "kube-api-access-rqktl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:22.307771 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.307571    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-dshm" pod "4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" (UID: "4aee1f03-06f6-485b-ac44-f60b7ba4ca6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:22.307874 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.307793    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-dshm" pod "3fff08e6-97c4-4572-9aad-95ae489fbffb" (UID: "3fff08e6-97c4-4572-9aad-95ae489fbffb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:22.308301 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.308266    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-kube-api-access-5rz9r" pod "4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" (UID: "4aee1f03-06f6-485b-ac44-f60b7ba4ca6b"). InnerVolumeSpecName "kube-api-access-5rz9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:22.404701 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.404667    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fff08e6-97c4-4572-9aad-95ae489fbffb-dshm\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:22.404701 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.404702    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rz9r\" (UniqueName: \"kubernetes.io/projected/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-kube-api-access-5rz9r\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:22.404923 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.404716    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-dshm\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:22.404923 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.404730    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fff08e6-97c4-4572-9aad-95ae489fbffb-tls-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:22.404923 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.404743    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-model-cache\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:22.404923 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.404754    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqktl\" (UniqueName: \"kubernetes.io/projected/3fff08e6-97c4-4572-9aad-95ae489fbffb-kube-api-access-rqktl\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:22.404923 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.404766    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b-tls-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:22.565608 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.565575    5958 generic.go:356] "Generic (PLEG): container finished" podID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerID="8518e28d69355a3546b16bbd82191ffbc89666665a0ee210293d7f14123630ba" exitCode=0
Oct 30 20:28:22.565792 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.565617    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"
Oct 30 20:28:22.565792 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.565648    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" event={"ID":"3fff08e6-97c4-4572-9aad-95ae489fbffb","Type":"ContainerDied","Data":"8518e28d69355a3546b16bbd82191ffbc89666665a0ee210293d7f14123630ba"}
Oct 30 20:28:22.565792 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.565692    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl" event={"ID":"3fff08e6-97c4-4572-9aad-95ae489fbffb","Type":"ContainerDied","Data":"efa39610e8fa9638240014b08abc7b80ed09e512c3abd139f9a130c6daef116c"}
Oct 30 20:28:22.565792 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.565709    5958 scope.go:122] "RemoveContainer" containerID="8518e28d69355a3546b16bbd82191ffbc89666665a0ee210293d7f14123630ba"
Oct 30 20:28:22.567306 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.567283    5958 generic.go:356] "Generic (PLEG): container finished" podID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerID="8cc6ee3c0a5a70aab4aaeba2a4a86f5545e39cd420ff5909029fcd255e24dc94" exitCode=0
Oct 30 20:28:22.567435 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.567320    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"
Oct 30 20:28:22.567435 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.567313    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" event={"ID":"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b","Type":"ContainerDied","Data":"8cc6ee3c0a5a70aab4aaeba2a4a86f5545e39cd420ff5909029fcd255e24dc94"}
Oct 30 20:28:22.567435 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.567378    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k" event={"ID":"4aee1f03-06f6-485b-ac44-f60b7ba4ca6b","Type":"ContainerDied","Data":"9d16046da8b75d87322cab986147d6964f75696d123aec5868b88bdd6ac60d45"}
Oct 30 20:28:22.575736 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.575718    5958 scope.go:122] "RemoveContainer" containerID="8518e28d69355a3546b16bbd82191ffbc89666665a0ee210293d7f14123630ba"
Oct 30 20:28:22.576016 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:28:22.575996    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8518e28d69355a3546b16bbd82191ffbc89666665a0ee210293d7f14123630ba\": container with ID starting with 8518e28d69355a3546b16bbd82191ffbc89666665a0ee210293d7f14123630ba not found: ID does not exist" containerID="8518e28d69355a3546b16bbd82191ffbc89666665a0ee210293d7f14123630ba"
Oct 30 20:28:22.576104 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.576027    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8518e28d69355a3546b16bbd82191ffbc89666665a0ee210293d7f14123630ba"} err="failed to get container status \"8518e28d69355a3546b16bbd82191ffbc89666665a0ee210293d7f14123630ba\": rpc error: code = NotFound desc = could not find container \"8518e28d69355a3546b16bbd82191ffbc89666665a0ee210293d7f14123630ba\": container with ID starting with 8518e28d69355a3546b16bbd82191ffbc89666665a0ee210293d7f14123630ba not found: ID does not exist"
Oct 30 20:28:22.576104 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.576050    5958 scope.go:122] "RemoveContainer" containerID="8cc6ee3c0a5a70aab4aaeba2a4a86f5545e39cd420ff5909029fcd255e24dc94"
Oct 30 20:28:22.586422 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.585165    5958 scope.go:122] "RemoveContainer" containerID="8cc6ee3c0a5a70aab4aaeba2a4a86f5545e39cd420ff5909029fcd255e24dc94"
Oct 30 20:28:22.586422 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:28:22.585629    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc6ee3c0a5a70aab4aaeba2a4a86f5545e39cd420ff5909029fcd255e24dc94\": container with ID starting with 8cc6ee3c0a5a70aab4aaeba2a4a86f5545e39cd420ff5909029fcd255e24dc94 not found: ID does not exist" containerID="8cc6ee3c0a5a70aab4aaeba2a4a86f5545e39cd420ff5909029fcd255e24dc94"
Oct 30 20:28:22.586422 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.585666    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc6ee3c0a5a70aab4aaeba2a4a86f5545e39cd420ff5909029fcd255e24dc94"} err="failed to get container status \"8cc6ee3c0a5a70aab4aaeba2a4a86f5545e39cd420ff5909029fcd255e24dc94\": rpc error: code = NotFound desc = could not find container \"8cc6ee3c0a5a70aab4aaeba2a4a86f5545e39cd420ff5909029fcd255e24dc94\": container with ID starting with 8cc6ee3c0a5a70aab4aaeba2a4a86f5545e39cd420ff5909029fcd255e24dc94 not found: ID does not exist"
Oct 30 20:28:22.732842 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.732802    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"]
Oct 30 20:28:22.735950 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.735923    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-779921/simulator-w0-779921-kserve-58f5755476-grz6k"]
Oct 30 20:28:22.844566 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.844488    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"]
Oct 30 20:28:22.846424 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:22.846397    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-779921/distinct-2-w0-779921-kserve-58f78dfb47-vmpsl"]
Oct 30 20:28:23.322108 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:23.322065    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns" podUID="58884900-a56f-40e1-8ce3-aea388a61245" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:28:23.322573 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:23.322150    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv" podUID="e0d98ef7-ed5f-4f06-b7f8-0ff074f41033" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:28:23.398581 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:23.398548    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" path="/var/lib/kubelet/pods/3fff08e6-97c4-4572-9aad-95ae489fbffb/volumes"
Oct 30 20:28:23.399008 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:23.398995    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" path="/var/lib/kubelet/pods/4aee1f03-06f6-485b-ac44-f60b7ba4ca6b/volumes"
Oct 30 20:28:32.366768 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:32.366735    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"]
Oct 30 20:28:33.321703 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:33.321664    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns" podUID="58884900-a56f-40e1-8ce3-aea388a61245" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:28:33.321884 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:33.321672    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv" podUID="e0d98ef7-ed5f-4f06-b7f8-0ff074f41033" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:28:36.606902 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:28:36.606851    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:28:39.208726 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.208697    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv"
Oct 30 20:28:39.302289 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.302257    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0d98ef7-ed5f-4f06-b7f8-0ff074f41033-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d98ef7-ed5f-4f06-b7f8-0ff074f41033-plugins-config-volume\") pod \"e0d98ef7-ed5f-4f06-b7f8-0ff074f41033\" (UID: \"e0d98ef7-ed5f-4f06-b7f8-0ff074f41033\") "
Oct 30 20:28:39.302485 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.302356    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d98ef7-ed5f-4f06-b7f8-0ff074f41033-kube-api-access-2vrdm\" (UniqueName: \"kubernetes.io/projected/e0d98ef7-ed5f-4f06-b7f8-0ff074f41033-kube-api-access-2vrdm\") pod \"e0d98ef7-ed5f-4f06-b7f8-0ff074f41033\" (UID: \"e0d98ef7-ed5f-4f06-b7f8-0ff074f41033\") "
Oct 30 20:28:39.302717 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.302691    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d98ef7-ed5f-4f06-b7f8-0ff074f41033-plugins-config-volume" pod "e0d98ef7-ed5f-4f06-b7f8-0ff074f41033" (UID: "e0d98ef7-ed5f-4f06-b7f8-0ff074f41033"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:28:39.302981 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.302963    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d98ef7-ed5f-4f06-b7f8-0ff074f41033-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:39.304785 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.304761    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d98ef7-ed5f-4f06-b7f8-0ff074f41033-kube-api-access-2vrdm" pod "e0d98ef7-ed5f-4f06-b7f8-0ff074f41033" (UID: "e0d98ef7-ed5f-4f06-b7f8-0ff074f41033"). InnerVolumeSpecName "kube-api-access-2vrdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:39.315118 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.315095    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns"
Oct 30 20:28:39.403372 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.403327    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58884900-a56f-40e1-8ce3-aea388a61245-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58884900-a56f-40e1-8ce3-aea388a61245-plugins-config-volume\") pod \"58884900-a56f-40e1-8ce3-aea388a61245\" (UID: \"58884900-a56f-40e1-8ce3-aea388a61245\") "
Oct 30 20:28:39.403565 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.403516    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58884900-a56f-40e1-8ce3-aea388a61245-kube-api-access-ks89p\" (UniqueName: \"kubernetes.io/projected/58884900-a56f-40e1-8ce3-aea388a61245-kube-api-access-ks89p\") pod \"58884900-a56f-40e1-8ce3-aea388a61245\" (UID: \"58884900-a56f-40e1-8ce3-aea388a61245\") "
Oct 30 20:28:39.403822 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.403795    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58884900-a56f-40e1-8ce3-aea388a61245-plugins-config-volume" pod "58884900-a56f-40e1-8ce3-aea388a61245" (UID: "58884900-a56f-40e1-8ce3-aea388a61245"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:28:39.404076 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.404053    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vrdm\" (UniqueName: \"kubernetes.io/projected/e0d98ef7-ed5f-4f06-b7f8-0ff074f41033-kube-api-access-2vrdm\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:39.404197 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.404078    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58884900-a56f-40e1-8ce3-aea388a61245-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:39.405717 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.405696    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58884900-a56f-40e1-8ce3-aea388a61245-kube-api-access-ks89p" pod "58884900-a56f-40e1-8ce3-aea388a61245" (UID: "58884900-a56f-40e1-8ce3-aea388a61245"). InnerVolumeSpecName "kube-api-access-ks89p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:39.504915 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.504821    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks89p\" (UniqueName: \"kubernetes.io/projected/58884900-a56f-40e1-8ce3-aea388a61245-kube-api-access-ks89p\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:39.674434 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.674393    5958 generic.go:356] "Generic (PLEG): container finished" podID="e0d98ef7-ed5f-4f06-b7f8-0ff074f41033" containerID="b2f7158ac1b49f7e5189c0d4f2f97b742369b8db0533d029446313106c15f0b0" exitCode=0
Oct 30 20:28:39.674640 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.674442    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv"
Oct 30 20:28:39.674640 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.674479    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv" event={"ID":"e0d98ef7-ed5f-4f06-b7f8-0ff074f41033","Type":"ContainerDied","Data":"b2f7158ac1b49f7e5189c0d4f2f97b742369b8db0533d029446313106c15f0b0"}
Oct 30 20:28:39.674640 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.674538    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv" event={"ID":"e0d98ef7-ed5f-4f06-b7f8-0ff074f41033","Type":"ContainerDied","Data":"d1c7b6146b035906699a4647d0151882cb24a26f0ff65031ad9b5a865fe0fc84"}
Oct 30 20:28:39.674640 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.674561    5958 scope.go:122] "RemoveContainer" containerID="b2f7158ac1b49f7e5189c0d4f2f97b742369b8db0533d029446313106c15f0b0"
Oct 30 20:28:39.676181 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.676157    5958 generic.go:356] "Generic (PLEG): container finished" podID="58884900-a56f-40e1-8ce3-aea388a61245" containerID="a19c64ec02ca84415ca31e7962bc489858e2308f2a30a6f21319ff19f1c3e0a5" exitCode=0
Oct 30 20:28:39.676508 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.676238    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns" event={"ID":"58884900-a56f-40e1-8ce3-aea388a61245","Type":"ContainerDied","Data":"a19c64ec02ca84415ca31e7962bc489858e2308f2a30a6f21319ff19f1c3e0a5"}
Oct 30 20:28:39.676508 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.676264    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns"
Oct 30 20:28:39.676508 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.676302    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns" event={"ID":"58884900-a56f-40e1-8ce3-aea388a61245","Type":"ContainerDied","Data":"eecbd3f82df975120064b5221dbfad13287e0abeda1c080cb4a79f8fefa65efb"}
Oct 30 20:28:39.686399 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.686224    5958 scope.go:122] "RemoveContainer" containerID="b2f7158ac1b49f7e5189c0d4f2f97b742369b8db0533d029446313106c15f0b0"
Oct 30 20:28:39.686661 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:28:39.686627    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f7158ac1b49f7e5189c0d4f2f97b742369b8db0533d029446313106c15f0b0\": container with ID starting with b2f7158ac1b49f7e5189c0d4f2f97b742369b8db0533d029446313106c15f0b0 not found: ID does not exist" containerID="b2f7158ac1b49f7e5189c0d4f2f97b742369b8db0533d029446313106c15f0b0"
Oct 30 20:28:39.686777 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.686662    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f7158ac1b49f7e5189c0d4f2f97b742369b8db0533d029446313106c15f0b0"} err="failed to get container status \"b2f7158ac1b49f7e5189c0d4f2f97b742369b8db0533d029446313106c15f0b0\": rpc error: code = NotFound desc = could not find container \"b2f7158ac1b49f7e5189c0d4f2f97b742369b8db0533d029446313106c15f0b0\": container with ID starting with b2f7158ac1b49f7e5189c0d4f2f97b742369b8db0533d029446313106c15f0b0 not found: ID does not exist"
Oct 30 20:28:39.686777 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.686688    5958 scope.go:122] "RemoveContainer" containerID="a19c64ec02ca84415ca31e7962bc489858e2308f2a30a6f21319ff19f1c3e0a5"
Oct 30 20:28:39.695762 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.695738    5958 scope.go:122] "RemoveContainer" containerID="a19c64ec02ca84415ca31e7962bc489858e2308f2a30a6f21319ff19f1c3e0a5"
Oct 30 20:28:39.696042 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:28:39.696016    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a19c64ec02ca84415ca31e7962bc489858e2308f2a30a6f21319ff19f1c3e0a5\": container with ID starting with a19c64ec02ca84415ca31e7962bc489858e2308f2a30a6f21319ff19f1c3e0a5 not found: ID does not exist" containerID="a19c64ec02ca84415ca31e7962bc489858e2308f2a30a6f21319ff19f1c3e0a5"
Oct 30 20:28:39.696124 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.696047    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19c64ec02ca84415ca31e7962bc489858e2308f2a30a6f21319ff19f1c3e0a5"} err="failed to get container status \"a19c64ec02ca84415ca31e7962bc489858e2308f2a30a6f21319ff19f1c3e0a5\": rpc error: code = NotFound desc = could not find container \"a19c64ec02ca84415ca31e7962bc489858e2308f2a30a6f21319ff19f1c3e0a5\": container with ID starting with a19c64ec02ca84415ca31e7962bc489858e2308f2a30a6f21319ff19f1c3e0a5 not found: ID does not exist"
Oct 30 20:28:39.851313 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.851278    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns"]
Oct 30 20:28:39.899049 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.899009    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-779921-7d9f866c8c-l4xns"]
Oct 30 20:28:39.975096 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.975066    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv"]
Oct 30 20:28:39.981462 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:39.981424    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-779921-c78c5f7d8-2tjxv"]
Oct 30 20:28:41.399130 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:41.399092    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58884900-a56f-40e1-8ce3-aea388a61245" path="/var/lib/kubelet/pods/58884900-a56f-40e1-8ce3-aea388a61245/volumes"
Oct 30 20:28:41.399738 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:41.399716    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0d98ef7-ed5f-4f06-b7f8-0ff074f41033" path="/var/lib/kubelet/pods/e0d98ef7-ed5f-4f06-b7f8-0ff074f41033/volumes"
Oct 30 20:28:42.345746 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.345709    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb1759f-2265d"]
Oct 30 20:28:42.346269 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.346254    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main"
Oct 30 20:28:42.346318 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.346270    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main"
Oct 30 20:28:42.346318 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.346295    5958 cpu_manager.go:414] "containerMap: removing container" podUID="b32f150b-7845-48de-8c12-61a7c1126976" containerName="main"
Oct 30 20:28:42.346318 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.346301    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b32f150b-7845-48de-8c12-61a7c1126976" containerName="main"
Oct 30 20:28:42.346318 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.346308    5958 cpu_manager.go:414] "containerMap: removing container" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main"
Oct 30 20:28:42.346318 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.346313    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main"
Oct 30 20:28:42.346318 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.346318    5958 cpu_manager.go:414] "containerMap: removing container" podUID="9b26180f-b498-4cc0-97e0-4b906a22f2f6" containerName="revoke-keys"
Oct 30 20:28:42.346557 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.346323    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b26180f-b498-4cc0-97e0-4b906a22f2f6" containerName="revoke-keys"
Oct 30 20:28:42.346557 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.346338    5958 cpu_manager.go:414] "containerMap: removing container" podUID="58884900-a56f-40e1-8ce3-aea388a61245" containerName="payload-pre-processing"
Oct 30 20:28:42.346557 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.346343    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58884900-a56f-40e1-8ce3-aea388a61245" containerName="payload-pre-processing"
Oct 30 20:28:42.346557 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.346350    5958 cpu_manager.go:414] "containerMap: removing container" podUID="04b5051c-1bff-4550-b442-7f0b7622a271" containerName="istio-proxy"
Oct 30 20:28:42.346557 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.346354    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b5051c-1bff-4550-b442-7f0b7622a271" containerName="istio-proxy"
Oct 30 20:28:42.346557 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.346362    5958 cpu_manager.go:414] "containerMap: removing container" podUID="e0d98ef7-ed5f-4f06-b7f8-0ff074f41033" containerName="payload-processing"
Oct 30 20:28:42.346557 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.346367    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0d98ef7-ed5f-4f06-b7f8-0ff074f41033" containerName="payload-processing"
Oct 30 20:28:42.346557 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.346382    5958 cpu_manager.go:414] "containerMap: removing container" podUID="f57472cf-a3f7-486d-9368-1c692e31bedc" containerName="payload-processing"
Oct 30 20:28:42.346557 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.346386    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f57472cf-a3f7-486d-9368-1c692e31bedc" containerName="payload-processing"
Oct 30 20:28:42.346557 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.346492    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="04b5051c-1bff-4550-b442-7f0b7622a271" containerName="istio-proxy"
Oct 30 20:28:42.346557 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.346504    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="f57472cf-a3f7-486d-9368-1c692e31bedc" containerName="payload-processing"
Oct 30 20:28:42.346557 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.346514    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fff08e6-97c4-4572-9aad-95ae489fbffb" containerName="main"
Oct 30 20:28:42.346557 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.346525    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="b32f150b-7845-48de-8c12-61a7c1126976" containerName="main"
Oct 30 20:28:42.346557 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.346531    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aee1f03-06f6-485b-ac44-f60b7ba4ca6b" containerName="main"
Oct 30 20:28:42.346557 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.346536    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b26180f-b498-4cc0-97e0-4b906a22f2f6" containerName="revoke-keys"
Oct 30 20:28:42.346557 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.346545    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="58884900-a56f-40e1-8ce3-aea388a61245" containerName="payload-pre-processing"
Oct 30 20:28:42.346557 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.346550    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="04b5051c-1bff-4550-b442-7f0b7622a271" containerName="istio-proxy"
Oct 30 20:28:42.346557 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.346557    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0d98ef7-ed5f-4f06-b7f8-0ff074f41033" containerName="payload-processing"
Oct 30 20:28:42.351744 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.351707    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb1759f-2265d"
Oct 30 20:28:42.408428 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.408390    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb1759f-2265d"]
Oct 30 20:28:42.454761 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.454725    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:28:42.454761 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.454731    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:28:42.456402 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.456355    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88103bf5-97bb-4a35-8906-51d55325f657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcb1759f-2265d\" (UID: \"88103bf5-97bb-4a35-8906-51d55325f657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb1759f-2265d"
Oct 30 20:28:42.456568 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.456473    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnqb6\" (UniqueName: \"kubernetes.io/projected/88103bf5-97bb-4a35-8906-51d55325f657-kube-api-access-lnqb6\") pod \"maas-api-revoke-sub-dcb1759f-2265d\" (UID: \"88103bf5-97bb-4a35-8906-51d55325f657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb1759f-2265d"
Oct 30 20:28:42.456568 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.456545    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88103bf5-97bb-4a35-8906-51d55325f657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcb1759f-2265d\" (UID: \"88103bf5-97bb-4a35-8906-51d55325f657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb1759f-2265d"
Oct 30 20:28:42.558094 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.558052    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88103bf5-97bb-4a35-8906-51d55325f657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcb1759f-2265d\" (UID: \"88103bf5-97bb-4a35-8906-51d55325f657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb1759f-2265d"
Oct 30 20:28:42.558273 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.558148    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnqb6\" (UniqueName: \"kubernetes.io/projected/88103bf5-97bb-4a35-8906-51d55325f657-kube-api-access-lnqb6\") pod \"maas-api-revoke-sub-dcb1759f-2265d\" (UID: \"88103bf5-97bb-4a35-8906-51d55325f657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb1759f-2265d"
Oct 30 20:28:42.558273 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.558202    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88103bf5-97bb-4a35-8906-51d55325f657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcb1759f-2265d\" (UID: \"88103bf5-97bb-4a35-8906-51d55325f657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb1759f-2265d"
Oct 30 20:28:42.559005 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.558910    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88103bf5-97bb-4a35-8906-51d55325f657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcb1759f-2265d\" (UID: \"88103bf5-97bb-4a35-8906-51d55325f657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb1759f-2265d"
Oct 30 20:28:42.560681 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.560660    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88103bf5-97bb-4a35-8906-51d55325f657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcb1759f-2265d\" (UID: \"88103bf5-97bb-4a35-8906-51d55325f657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb1759f-2265d"
Oct 30 20:28:42.578504 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.578434    5958 scope.go:122] "RemoveContainer" containerID="9e374ead97cf5a8f0e4fedbb1289c4b2d9f7d650b84fa83c4bff48b5c2e2be23"
Oct 30 20:28:42.619974 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.619946    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnqb6\" (UniqueName: \"kubernetes.io/projected/88103bf5-97bb-4a35-8906-51d55325f657-kube-api-access-lnqb6\") pod \"maas-api-revoke-sub-dcb1759f-2265d\" (UID: \"88103bf5-97bb-4a35-8906-51d55325f657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb1759f-2265d"
Oct 30 20:28:42.666614 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.666578    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb1759f-2265d"
Oct 30 20:28:42.749031 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.748995    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc"]
Oct 30 20:28:42.863219 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:42.863136    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d"]
Oct 30 20:28:43.042225 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:43.042200    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb1759f-2265d"]
Oct 30 20:28:43.050636 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:28:43.050604    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88103bf5_97bb_4a35_8906_51d55325f657.slice/crio-a9ee9c18cfc2531e61b140bd20727ba04b9a1602eac807abeaa760388d827d38 WatchSource:0}: Error finding container a9ee9c18cfc2531e61b140bd20727ba04b9a1602eac807abeaa760388d827d38: Status 404 returned error can't find the container with id a9ee9c18cfc2531e61b140bd20727ba04b9a1602eac807abeaa760388d827d38
Oct 30 20:28:43.705231 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:43.705195    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb1759f-2265d" event={"ID":"88103bf5-97bb-4a35-8906-51d55325f657","Type":"ContainerStarted","Data":"60290760dceb9c7c8c1fc7a8f9c7c448b5d5d23221df08b8a3751bc4f079cbb7"}
Oct 30 20:28:43.705693 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:43.705248    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb1759f-2265d" event={"ID":"88103bf5-97bb-4a35-8906-51d55325f657","Type":"ContainerStarted","Data":"a9ee9c18cfc2531e61b140bd20727ba04b9a1602eac807abeaa760388d827d38"}
Oct 30 20:28:43.823486 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:43.823415    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb1759f-2265d" podStartSLOduration=1.8233991870000001 podStartE2EDuration="1.823399187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:28:43.820753775 +0000 UTC m=+2762.552969177" watchObservedRunningTime="2026-10-30 20:28:43.823399187 +0000 UTC m=+2762.555614575"
Oct 30 20:28:44.712129 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:44.712090    5958 generic.go:356] "Generic (PLEG): container finished" podID="88103bf5-97bb-4a35-8906-51d55325f657" containerID="60290760dceb9c7c8c1fc7a8f9c7c448b5d5d23221df08b8a3751bc4f079cbb7" exitCode=0
Oct 30 20:28:44.712559 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:44.712177    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb1759f-2265d" event={"ID":"88103bf5-97bb-4a35-8906-51d55325f657","Type":"ContainerDied","Data":"60290760dceb9c7c8c1fc7a8f9c7c448b5d5d23221df08b8a3751bc4f079cbb7"}
Oct 30 20:28:45.890790 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:45.890752    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb1759f-2265d"
Oct 30 20:28:45.996185 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:45.996097    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88103bf5-97bb-4a35-8906-51d55325f657-kube-api-access-lnqb6\" (UniqueName: \"kubernetes.io/projected/88103bf5-97bb-4a35-8906-51d55325f657-kube-api-access-lnqb6\") pod \"88103bf5-97bb-4a35-8906-51d55325f657\" (UID: \"88103bf5-97bb-4a35-8906-51d55325f657\") "
Oct 30 20:28:45.996356 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:45.996210    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88103bf5-97bb-4a35-8906-51d55325f657-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88103bf5-97bb-4a35-8906-51d55325f657-maas-api-openshift-ca\") pod \"88103bf5-97bb-4a35-8906-51d55325f657\" (UID: \"88103bf5-97bb-4a35-8906-51d55325f657\") "
Oct 30 20:28:45.996356 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:45.996261    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88103bf5-97bb-4a35-8906-51d55325f657-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88103bf5-97bb-4a35-8906-51d55325f657-maas-api-xks-ca\") pod \"88103bf5-97bb-4a35-8906-51d55325f657\" (UID: \"88103bf5-97bb-4a35-8906-51d55325f657\") "
Oct 30 20:28:45.996611 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:45.996574    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88103bf5-97bb-4a35-8906-51d55325f657-maas-api-openshift-ca" pod "88103bf5-97bb-4a35-8906-51d55325f657" (UID: "88103bf5-97bb-4a35-8906-51d55325f657"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:28:45.998554 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:45.998529    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88103bf5-97bb-4a35-8906-51d55325f657-maas-api-xks-ca" pod "88103bf5-97bb-4a35-8906-51d55325f657" (UID: "88103bf5-97bb-4a35-8906-51d55325f657"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:28:45.998554 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:45.998540    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88103bf5-97bb-4a35-8906-51d55325f657-kube-api-access-lnqb6" pod "88103bf5-97bb-4a35-8906-51d55325f657" (UID: "88103bf5-97bb-4a35-8906-51d55325f657"). InnerVolumeSpecName "kube-api-access-lnqb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:46.098235 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:46.098194    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88103bf5-97bb-4a35-8906-51d55325f657-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:46.098235 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:46.098231    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnqb6\" (UniqueName: \"kubernetes.io/projected/88103bf5-97bb-4a35-8906-51d55325f657-kube-api-access-lnqb6\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:46.098235 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:46.098242    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88103bf5-97bb-4a35-8906-51d55325f657-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:46.724953 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:46.724918    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb1759f-2265d"
Oct 30 20:28:46.724953 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:46.724944    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb1759f-2265d" event={"ID":"88103bf5-97bb-4a35-8906-51d55325f657","Type":"ContainerDied","Data":"a9ee9c18cfc2531e61b140bd20727ba04b9a1602eac807abeaa760388d827d38"}
Oct 30 20:28:46.725176 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:46.724990    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9ee9c18cfc2531e61b140bd20727ba04b9a1602eac807abeaa760388d827d38"
Oct 30 20:28:47.387006 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.386959    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw" podUID="5e07ac50-c692-49c7-a2c3-dcbc364693ae" containerName="main" containerID="cri-o://fcc9816739a27284a13c6eb3ae13350efb453913ed806458b5d123e5ed3fbc6d" gracePeriod=45
Oct 30 20:28:47.449162 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.449108    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw" podUID="5e07ac50-c692-49c7-a2c3-dcbc364693ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.156:8000/ready\": dial tcp 10.134.0.156:8000: connect: connection refused"
Oct 30 20:28:47.664267 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.664240    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:28:47.732328 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.732296    5958 generic.go:356] "Generic (PLEG): container finished" podID="5e07ac50-c692-49c7-a2c3-dcbc364693ae" containerID="fcc9816739a27284a13c6eb3ae13350efb453913ed806458b5d123e5ed3fbc6d" exitCode=0
Oct 30 20:28:47.732519 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.732337    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"
Oct 30 20:28:47.732519 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.732363    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw" event={"ID":"5e07ac50-c692-49c7-a2c3-dcbc364693ae","Type":"ContainerDied","Data":"fcc9816739a27284a13c6eb3ae13350efb453913ed806458b5d123e5ed3fbc6d"}
Oct 30 20:28:47.732519 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.732401    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw" event={"ID":"5e07ac50-c692-49c7-a2c3-dcbc364693ae","Type":"ContainerDied","Data":"7085986f97c0c57b9cdbda33a55679c0555530a1367b911d0e4b4b3c4b4b83fc"}
Oct 30 20:28:47.732519 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.732417    5958 scope.go:122] "RemoveContainer" containerID="fcc9816739a27284a13c6eb3ae13350efb453913ed806458b5d123e5ed3fbc6d"
Oct 30 20:28:47.755674 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.755629    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc" podUID="7578c703-e969-495b-98e9-70bfdf1f8413" containerName="payload-processing" containerID="cri-o://4bb5fa5fc5771b0eaac80e7b62f4d2b00c0fe00ea18bbf0738dac21aa84d6122" gracePeriod=25
Oct 30 20:28:47.755955 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.755701    5958 scope.go:122] "RemoveContainer" containerID="fcc9816739a27284a13c6eb3ae13350efb453913ed806458b5d123e5ed3fbc6d"
Oct 30 20:28:47.756823 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:28:47.756791    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc9816739a27284a13c6eb3ae13350efb453913ed806458b5d123e5ed3fbc6d\": container with ID starting with fcc9816739a27284a13c6eb3ae13350efb453913ed806458b5d123e5ed3fbc6d not found: ID does not exist" containerID="fcc9816739a27284a13c6eb3ae13350efb453913ed806458b5d123e5ed3fbc6d"
Oct 30 20:28:47.756942 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.756833    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc9816739a27284a13c6eb3ae13350efb453913ed806458b5d123e5ed3fbc6d"} err="failed to get container status \"fcc9816739a27284a13c6eb3ae13350efb453913ed806458b5d123e5ed3fbc6d\": rpc error: code = NotFound desc = could not find container \"fcc9816739a27284a13c6eb3ae13350efb453913ed806458b5d123e5ed3fbc6d\": container with ID starting with fcc9816739a27284a13c6eb3ae13350efb453913ed806458b5d123e5ed3fbc6d not found: ID does not exist"
Oct 30 20:28:47.818506 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.818474    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e07ac50-c692-49c7-a2c3-dcbc364693ae-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e07ac50-c692-49c7-a2c3-dcbc364693ae-tls-certs\") pod \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\" (UID: \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\") "
Oct 30 20:28:47.818688 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.818527    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-model-cache\") pod \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\" (UID: \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\") "
Oct 30 20:28:47.818688 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.818574    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-tmp-dir\") pod \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\" (UID: \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\") "
Oct 30 20:28:47.818813 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.818711    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-home\" (UniqueName: \"kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-home\") pod \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\" (UID: \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\") "
Oct 30 20:28:47.818813 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.818773    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-dshm\") pod \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\" (UID: \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\") "
Oct 30 20:28:47.818909 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.818849    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-model-cache" pod "5e07ac50-c692-49c7-a2c3-dcbc364693ae" (UID: "5e07ac50-c692-49c7-a2c3-dcbc364693ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:47.818909 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.818869    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e07ac50-c692-49c7-a2c3-dcbc364693ae-kube-api-access-h4lj6\" (UniqueName: \"kubernetes.io/projected/5e07ac50-c692-49c7-a2c3-dcbc364693ae-kube-api-access-h4lj6\") pod \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\" (UID: \"5e07ac50-c692-49c7-a2c3-dcbc364693ae\") "
Oct 30 20:28:47.818909 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.818888    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-home" pod "5e07ac50-c692-49c7-a2c3-dcbc364693ae" (UID: "5e07ac50-c692-49c7-a2c3-dcbc364693ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:47.819121 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.819100    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-tmp-dir" pod "5e07ac50-c692-49c7-a2c3-dcbc364693ae" (UID: "5e07ac50-c692-49c7-a2c3-dcbc364693ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:47.819354 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.819316    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-home\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:47.819354 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.819352    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-model-cache\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:47.819622 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.819367    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-tmp-dir\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:47.821424 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.821392    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e07ac50-c692-49c7-a2c3-dcbc364693ae-kube-api-access-h4lj6" pod "5e07ac50-c692-49c7-a2c3-dcbc364693ae" (UID: "5e07ac50-c692-49c7-a2c3-dcbc364693ae"). InnerVolumeSpecName "kube-api-access-h4lj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:47.821560 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.821424    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-dshm" pod "5e07ac50-c692-49c7-a2c3-dcbc364693ae" (UID: "5e07ac50-c692-49c7-a2c3-dcbc364693ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:28:47.821699 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.821677    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e07ac50-c692-49c7-a2c3-dcbc364693ae-tls-certs" pod "5e07ac50-c692-49c7-a2c3-dcbc364693ae" (UID: "5e07ac50-c692-49c7-a2c3-dcbc364693ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:28:47.863874 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.863836    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d" podUID="3224f5ec-0f50-454e-a07d-e16c0b9fe71b" containerName="payload-pre-processing" containerID="cri-o://c60700ba8f7be5d773de166e95b86211d89ee969e1a7c6e58091f6c8860e9061" gracePeriod=25
Oct 30 20:28:47.920850 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.920766    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4lj6\" (UniqueName: \"kubernetes.io/projected/5e07ac50-c692-49c7-a2c3-dcbc364693ae-kube-api-access-h4lj6\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:47.920850 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.920806    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e07ac50-c692-49c7-a2c3-dcbc364693ae-tls-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:47.920850 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.920816    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e07ac50-c692-49c7-a2c3-dcbc364693ae-dshm\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:47.991291 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:47.991263    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc"
Oct 30 20:28:48.122907 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:48.122869    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7578c703-e969-495b-98e9-70bfdf1f8413-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7578c703-e969-495b-98e9-70bfdf1f8413-plugins-config-volume\") pod \"7578c703-e969-495b-98e9-70bfdf1f8413\" (UID: \"7578c703-e969-495b-98e9-70bfdf1f8413\") "
Oct 30 20:28:48.123079 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:48.123040    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7578c703-e969-495b-98e9-70bfdf1f8413-kube-api-access-b5ssn\" (UniqueName: \"kubernetes.io/projected/7578c703-e969-495b-98e9-70bfdf1f8413-kube-api-access-b5ssn\") pod \"7578c703-e969-495b-98e9-70bfdf1f8413\" (UID: \"7578c703-e969-495b-98e9-70bfdf1f8413\") "
Oct 30 20:28:48.123307 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:48.123284    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7578c703-e969-495b-98e9-70bfdf1f8413-plugins-config-volume" pod "7578c703-e969-495b-98e9-70bfdf1f8413" (UID: "7578c703-e969-495b-98e9-70bfdf1f8413"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:28:48.123584 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:48.123563    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7578c703-e969-495b-98e9-70bfdf1f8413-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:48.125284 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:48.125259    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7578c703-e969-495b-98e9-70bfdf1f8413-kube-api-access-b5ssn" pod "7578c703-e969-495b-98e9-70bfdf1f8413" (UID: "7578c703-e969-495b-98e9-70bfdf1f8413"). InnerVolumeSpecName "kube-api-access-b5ssn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:48.221513 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:48.221406    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"]
Oct 30 20:28:48.223328 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:48.223305    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-453277/test-model-453277-kserve-df5f698dc-t5jfw"]
Oct 30 20:28:48.224254 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:48.224226    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5ssn\" (UniqueName: \"kubernetes.io/projected/7578c703-e969-495b-98e9-70bfdf1f8413-kube-api-access-b5ssn\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:28:48.740292 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:48.740257    5958 generic.go:356] "Generic (PLEG): container finished" podID="7578c703-e969-495b-98e9-70bfdf1f8413" containerID="4bb5fa5fc5771b0eaac80e7b62f4d2b00c0fe00ea18bbf0738dac21aa84d6122" exitCode=0
Oct 30 20:28:48.740800 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:48.740298    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc"
Oct 30 20:28:48.740800 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:48.740310    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc" event={"ID":"7578c703-e969-495b-98e9-70bfdf1f8413","Type":"ContainerDied","Data":"4bb5fa5fc5771b0eaac80e7b62f4d2b00c0fe00ea18bbf0738dac21aa84d6122"}
Oct 30 20:28:48.740800 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:48.740357    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc" event={"ID":"7578c703-e969-495b-98e9-70bfdf1f8413","Type":"ContainerDied","Data":"564ce05c3615bf8239e07b7f1a0aae2a7d4ce74e42404ac7a5350a7c2aad4004"}
Oct 30 20:28:48.740800 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:48.740382    5958 scope.go:122] "RemoveContainer" containerID="4bb5fa5fc5771b0eaac80e7b62f4d2b00c0fe00ea18bbf0738dac21aa84d6122"
Oct 30 20:28:48.750899 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:48.750873    5958 scope.go:122] "RemoveContainer" containerID="4bb5fa5fc5771b0eaac80e7b62f4d2b00c0fe00ea18bbf0738dac21aa84d6122"
Oct 30 20:28:48.751232 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:28:48.751213    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb5fa5fc5771b0eaac80e7b62f4d2b00c0fe00ea18bbf0738dac21aa84d6122\": container with ID starting with 4bb5fa5fc5771b0eaac80e7b62f4d2b00c0fe00ea18bbf0738dac21aa84d6122 not found: ID does not exist" containerID="4bb5fa5fc5771b0eaac80e7b62f4d2b00c0fe00ea18bbf0738dac21aa84d6122"
Oct 30 20:28:48.751289 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:48.751243    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb5fa5fc5771b0eaac80e7b62f4d2b00c0fe00ea18bbf0738dac21aa84d6122"} err="failed to get container status \"4bb5fa5fc5771b0eaac80e7b62f4d2b00c0fe00ea18bbf0738dac21aa84d6122\": rpc error: code = NotFound desc = could not find container \"4bb5fa5fc5771b0eaac80e7b62f4d2b00c0fe00ea18bbf0738dac21aa84d6122\": container with ID starting with 4bb5fa5fc5771b0eaac80e7b62f4d2b00c0fe00ea18bbf0738dac21aa84d6122 not found: ID does not exist"
Oct 30 20:28:48.905783 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:48.905742    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc"]
Oct 30 20:28:48.955834 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:48.955800    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-01e8dc-7bd67789db-wc8jc"]
Oct 30 20:28:49.398629 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:49.398595    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e07ac50-c692-49c7-a2c3-dcbc364693ae" path="/var/lib/kubelet/pods/5e07ac50-c692-49c7-a2c3-dcbc364693ae/volumes"
Oct 30 20:28:49.399062 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:49.399048    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7578c703-e969-495b-98e9-70bfdf1f8413" path="/var/lib/kubelet/pods/7578c703-e969-495b-98e9-70bfdf1f8413/volumes"
Oct 30 20:28:50.947740 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:28:50.947698    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d" podUID="3224f5ec-0f50-454e-a07d-e16c0b9fe71b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:29:00.947532 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:00.947485    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d" podUID="3224f5ec-0f50-454e-a07d-e16c0b9fe71b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:29:06.606719 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:29:06.606673    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:29:08.056171 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:08.056136    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4269370-x8n62"]
Oct 30 20:29:08.104895 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:08.104861    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4269370-x8n62"]
Oct 30 20:29:08.122072 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:08.122006    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d"
Oct 30 20:29:08.255947 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:08.255917    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3224f5ec-0f50-454e-a07d-e16c0b9fe71b-kube-api-access-r7t4c\" (UniqueName: \"kubernetes.io/projected/3224f5ec-0f50-454e-a07d-e16c0b9fe71b-kube-api-access-r7t4c\") pod \"3224f5ec-0f50-454e-a07d-e16c0b9fe71b\" (UID: \"3224f5ec-0f50-454e-a07d-e16c0b9fe71b\") "
Oct 30 20:29:08.256127 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:08.255969    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3224f5ec-0f50-454e-a07d-e16c0b9fe71b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3224f5ec-0f50-454e-a07d-e16c0b9fe71b-plugins-config-volume\") pod \"3224f5ec-0f50-454e-a07d-e16c0b9fe71b\" (UID: \"3224f5ec-0f50-454e-a07d-e16c0b9fe71b\") "
Oct 30 20:29:08.256460 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:08.256426    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3224f5ec-0f50-454e-a07d-e16c0b9fe71b-plugins-config-volume" pod "3224f5ec-0f50-454e-a07d-e16c0b9fe71b" (UID: "3224f5ec-0f50-454e-a07d-e16c0b9fe71b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:29:08.258324 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:08.258294    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3224f5ec-0f50-454e-a07d-e16c0b9fe71b-kube-api-access-r7t4c" pod "3224f5ec-0f50-454e-a07d-e16c0b9fe71b" (UID: "3224f5ec-0f50-454e-a07d-e16c0b9fe71b"). InnerVolumeSpecName "kube-api-access-r7t4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:08.357615 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:08.357584    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7t4c\" (UniqueName: \"kubernetes.io/projected/3224f5ec-0f50-454e-a07d-e16c0b9fe71b-kube-api-access-r7t4c\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:08.357615 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:08.357613    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3224f5ec-0f50-454e-a07d-e16c0b9fe71b-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:08.857393 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:08.857349    5958 generic.go:356] "Generic (PLEG): container finished" podID="3224f5ec-0f50-454e-a07d-e16c0b9fe71b" containerID="c60700ba8f7be5d773de166e95b86211d89ee969e1a7c6e58091f6c8860e9061" exitCode=0
Oct 30 20:29:08.857393 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:08.857391    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d" event={"ID":"3224f5ec-0f50-454e-a07d-e16c0b9fe71b","Type":"ContainerDied","Data":"c60700ba8f7be5d773de166e95b86211d89ee969e1a7c6e58091f6c8860e9061"}
Oct 30 20:29:08.857646 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:08.857417    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d"
Oct 30 20:29:08.857646 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:08.857427    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d" event={"ID":"3224f5ec-0f50-454e-a07d-e16c0b9fe71b","Type":"ContainerDied","Data":"45fd528a39e6b845dec863efb19bdcb0495ab961505e112502b5feee26af40d0"}
Oct 30 20:29:08.857646 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:08.857462    5958 scope.go:122] "RemoveContainer" containerID="c60700ba8f7be5d773de166e95b86211d89ee969e1a7c6e58091f6c8860e9061"
Oct 30 20:29:08.867536 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:08.867505    5958 scope.go:122] "RemoveContainer" containerID="c60700ba8f7be5d773de166e95b86211d89ee969e1a7c6e58091f6c8860e9061"
Oct 30 20:29:08.867846 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:29:08.867817    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c60700ba8f7be5d773de166e95b86211d89ee969e1a7c6e58091f6c8860e9061\": container with ID starting with c60700ba8f7be5d773de166e95b86211d89ee969e1a7c6e58091f6c8860e9061 not found: ID does not exist" containerID="c60700ba8f7be5d773de166e95b86211d89ee969e1a7c6e58091f6c8860e9061"
Oct 30 20:29:08.867909 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:08.867856    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c60700ba8f7be5d773de166e95b86211d89ee969e1a7c6e58091f6c8860e9061"} err="failed to get container status \"c60700ba8f7be5d773de166e95b86211d89ee969e1a7c6e58091f6c8860e9061\": rpc error: code = NotFound desc = could not find container \"c60700ba8f7be5d773de166e95b86211d89ee969e1a7c6e58091f6c8860e9061\": container with ID starting with c60700ba8f7be5d773de166e95b86211d89ee969e1a7c6e58091f6c8860e9061 not found: ID does not exist"
Oct 30 20:29:09.023978 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:09.023940    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d"]
Oct 30 20:29:09.027712 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:09.027682    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-01e8dc-6bf755c6f5-8bt4d"]
Oct 30 20:29:09.076501 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:09.076437    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a404308-mqk8z"]
Oct 30 20:29:09.078248 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:09.078221    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a404308-mqk8z"]
Oct 30 20:29:09.398627 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:09.398595    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3224f5ec-0f50-454e-a07d-e16c0b9fe71b" path="/var/lib/kubelet/pods/3224f5ec-0f50-454e-a07d-e16c0b9fe71b/volumes"
Oct 30 20:29:09.399046 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:09.399032    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2356c8f-e080-431d-90c2-71b838f7e1e8" path="/var/lib/kubelet/pods/d2356c8f-e080-431d-90c2-71b838f7e1e8/volumes"
Oct 30 20:29:09.399421 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:09.399407    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9bea5d8-aef0-4bf0-a76d-a584ce0088a9" path="/var/lib/kubelet/pods/f9bea5d8-aef0-4bf0-a76d-a584ce0088a9/volumes"
Oct 30 20:29:10.988969 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:10.988929    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm"]
Oct 30 20:29:11.093739 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:11.093698    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx"]
Oct 30 20:29:15.989666 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:15.989594    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm" podUID="60327e5a-6b34-4eff-bfaa-6af3715d0b78" containerName="payload-processing" containerID="cri-o://16946d3da87ea5268252d3104a208d4928f430331822ee23e87210dd2e0964ff" gracePeriod=25
Oct 30 20:29:16.094363 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:16.094290    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx" podUID="c9fa71b4-8a5b-40e6-be72-35583f6560f5" containerName="payload-pre-processing" containerID="cri-o://b6fbd421df27db7bb75bd7f6ed2cade3c2e2879354336cdb04fb46e4fe4e00ab" gracePeriod=25
Oct 30 20:29:16.254474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:16.254391    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm"
Oct 30 20:29:16.444480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:16.444410    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60327e5a-6b34-4eff-bfaa-6af3715d0b78-kube-api-access-pnfgf\" (UniqueName: \"kubernetes.io/projected/60327e5a-6b34-4eff-bfaa-6af3715d0b78-kube-api-access-pnfgf\") pod \"60327e5a-6b34-4eff-bfaa-6af3715d0b78\" (UID: \"60327e5a-6b34-4eff-bfaa-6af3715d0b78\") "
Oct 30 20:29:16.444676 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:16.444590    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60327e5a-6b34-4eff-bfaa-6af3715d0b78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60327e5a-6b34-4eff-bfaa-6af3715d0b78-plugins-config-volume\") pod \"60327e5a-6b34-4eff-bfaa-6af3715d0b78\" (UID: \"60327e5a-6b34-4eff-bfaa-6af3715d0b78\") "
Oct 30 20:29:16.444980 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:16.444957    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60327e5a-6b34-4eff-bfaa-6af3715d0b78-plugins-config-volume" pod "60327e5a-6b34-4eff-bfaa-6af3715d0b78" (UID: "60327e5a-6b34-4eff-bfaa-6af3715d0b78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:29:16.446850 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:16.446830    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60327e5a-6b34-4eff-bfaa-6af3715d0b78-kube-api-access-pnfgf" pod "60327e5a-6b34-4eff-bfaa-6af3715d0b78" (UID: "60327e5a-6b34-4eff-bfaa-6af3715d0b78"). InnerVolumeSpecName "kube-api-access-pnfgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:16.545960 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:16.545861    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60327e5a-6b34-4eff-bfaa-6af3715d0b78-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:16.545960 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:16.545893    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnfgf\" (UniqueName: \"kubernetes.io/projected/60327e5a-6b34-4eff-bfaa-6af3715d0b78-kube-api-access-pnfgf\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:16.903066 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:16.903031    5958 generic.go:356] "Generic (PLEG): container finished" podID="60327e5a-6b34-4eff-bfaa-6af3715d0b78" containerID="16946d3da87ea5268252d3104a208d4928f430331822ee23e87210dd2e0964ff" exitCode=0
Oct 30 20:29:16.903335 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:16.903096    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm"
Oct 30 20:29:16.903335 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:16.903106    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm" event={"ID":"60327e5a-6b34-4eff-bfaa-6af3715d0b78","Type":"ContainerDied","Data":"16946d3da87ea5268252d3104a208d4928f430331822ee23e87210dd2e0964ff"}
Oct 30 20:29:16.903335 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:16.903163    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm" event={"ID":"60327e5a-6b34-4eff-bfaa-6af3715d0b78","Type":"ContainerDied","Data":"bc43f18d02148ae0035c35e64d7973112f8f105192ef003ea148994057dc72c5"}
Oct 30 20:29:16.903335 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:16.903191    5958 scope.go:122] "RemoveContainer" containerID="16946d3da87ea5268252d3104a208d4928f430331822ee23e87210dd2e0964ff"
Oct 30 20:29:16.912826 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:16.912807    5958 scope.go:122] "RemoveContainer" containerID="16946d3da87ea5268252d3104a208d4928f430331822ee23e87210dd2e0964ff"
Oct 30 20:29:16.913096 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:29:16.913071    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16946d3da87ea5268252d3104a208d4928f430331822ee23e87210dd2e0964ff\": container with ID starting with 16946d3da87ea5268252d3104a208d4928f430331822ee23e87210dd2e0964ff not found: ID does not exist" containerID="16946d3da87ea5268252d3104a208d4928f430331822ee23e87210dd2e0964ff"
Oct 30 20:29:16.913162 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:16.913107    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16946d3da87ea5268252d3104a208d4928f430331822ee23e87210dd2e0964ff"} err="failed to get container status \"16946d3da87ea5268252d3104a208d4928f430331822ee23e87210dd2e0964ff\": rpc error: code = NotFound desc = could not find container \"16946d3da87ea5268252d3104a208d4928f430331822ee23e87210dd2e0964ff\": container with ID starting with 16946d3da87ea5268252d3104a208d4928f430331822ee23e87210dd2e0964ff not found: ID does not exist"
Oct 30 20:29:17.075502 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:17.075461    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm"]
Oct 30 20:29:17.080697 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:17.080669    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-453277-59bdbd6494-jk4gm"]
Oct 30 20:29:17.398043 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:17.398013    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60327e5a-6b34-4eff-bfaa-6af3715d0b78" path="/var/lib/kubelet/pods/60327e5a-6b34-4eff-bfaa-6af3715d0b78/volumes"
Oct 30 20:29:24.926377 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:24.926286    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h"]
Oct 30 20:29:24.926807 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:24.926542    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h" podUID="e12ca895-aa85-41db-89bf-d9912ba799f9" containerName="maas-api" containerID="cri-o://7bef49e364f7eb5f614947b48aa790585af70bba4029bfee7451e9f7b7494242" gracePeriod=30
Oct 30 20:29:25.211224 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:25.211193    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h"
Oct 30 20:29:25.232515 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:25.232423    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx" podUID="c9fa71b4-8a5b-40e6-be72-35583f6560f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:29:25.237658 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:25.237636    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e12ca895-aa85-41db-89bf-d9912ba799f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e12ca895-aa85-41db-89bf-d9912ba799f9-metrics-certs\") pod \"e12ca895-aa85-41db-89bf-d9912ba799f9\" (UID: \"e12ca895-aa85-41db-89bf-d9912ba799f9\") "
Oct 30 20:29:25.237753 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:25.237676    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e12ca895-aa85-41db-89bf-d9912ba799f9-kube-api-access-fmskn\" (UniqueName: \"kubernetes.io/projected/e12ca895-aa85-41db-89bf-d9912ba799f9-kube-api-access-fmskn\") pod \"e12ca895-aa85-41db-89bf-d9912ba799f9\" (UID: \"e12ca895-aa85-41db-89bf-d9912ba799f9\") "
Oct 30 20:29:25.237807 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:25.237793    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e12ca895-aa85-41db-89bf-d9912ba799f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12ca895-aa85-41db-89bf-d9912ba799f9-maas-api-tls\") pod \"e12ca895-aa85-41db-89bf-d9912ba799f9\" (UID: \"e12ca895-aa85-41db-89bf-d9912ba799f9\") "
Oct 30 20:29:25.237909 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:25.237894    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e12ca895-aa85-41db-89bf-d9912ba799f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12ca895-aa85-41db-89bf-d9912ba799f9-openshift-service-ca\") pod \"e12ca895-aa85-41db-89bf-d9912ba799f9\" (UID: \"e12ca895-aa85-41db-89bf-d9912ba799f9\") "
Oct 30 20:29:25.238262 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:25.238227    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e12ca895-aa85-41db-89bf-d9912ba799f9-openshift-service-ca" pod "e12ca895-aa85-41db-89bf-d9912ba799f9" (UID: "e12ca895-aa85-41db-89bf-d9912ba799f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:29:25.239911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:25.239887    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e12ca895-aa85-41db-89bf-d9912ba799f9-maas-api-tls" pod "e12ca895-aa85-41db-89bf-d9912ba799f9" (UID: "e12ca895-aa85-41db-89bf-d9912ba799f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:29:25.240005 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:25.239980    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e12ca895-aa85-41db-89bf-d9912ba799f9-metrics-certs" pod "e12ca895-aa85-41db-89bf-d9912ba799f9" (UID: "e12ca895-aa85-41db-89bf-d9912ba799f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:29:25.240005 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:25.239987    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12ca895-aa85-41db-89bf-d9912ba799f9-kube-api-access-fmskn" pod "e12ca895-aa85-41db-89bf-d9912ba799f9" (UID: "e12ca895-aa85-41db-89bf-d9912ba799f9"). InnerVolumeSpecName "kube-api-access-fmskn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:25.339537 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:25.339498    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12ca895-aa85-41db-89bf-d9912ba799f9-openshift-service-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:25.339537 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:25.339539    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e12ca895-aa85-41db-89bf-d9912ba799f9-metrics-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:25.339737 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:25.339554    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmskn\" (UniqueName: \"kubernetes.io/projected/e12ca895-aa85-41db-89bf-d9912ba799f9-kube-api-access-fmskn\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:25.339737 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:25.339568    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12ca895-aa85-41db-89bf-d9912ba799f9-maas-api-tls\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:25.960293 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:25.960253    5958 generic.go:356] "Generic (PLEG): container finished" podID="e12ca895-aa85-41db-89bf-d9912ba799f9" containerID="7bef49e364f7eb5f614947b48aa790585af70bba4029bfee7451e9f7b7494242" exitCode=0
Oct 30 20:29:25.960835 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:25.960306    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h"
Oct 30 20:29:25.960835 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:25.960344    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h" event={"ID":"e12ca895-aa85-41db-89bf-d9912ba799f9","Type":"ContainerDied","Data":"7bef49e364f7eb5f614947b48aa790585af70bba4029bfee7451e9f7b7494242"}
Oct 30 20:29:25.960835 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:25.960403    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h" event={"ID":"e12ca895-aa85-41db-89bf-d9912ba799f9","Type":"ContainerDied","Data":"204df3d867834dccefb1bd104c7a03e6771a53890bb5b6e0ec7c0157838cf367"}
Oct 30 20:29:25.960835 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:25.960416    5958 scope.go:122] "RemoveContainer" containerID="7bef49e364f7eb5f614947b48aa790585af70bba4029bfee7451e9f7b7494242"
Oct 30 20:29:25.969994 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:25.969970    5958 scope.go:122] "RemoveContainer" containerID="7bef49e364f7eb5f614947b48aa790585af70bba4029bfee7451e9f7b7494242"
Oct 30 20:29:25.970277 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:29:25.970257    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bef49e364f7eb5f614947b48aa790585af70bba4029bfee7451e9f7b7494242\": container with ID starting with 7bef49e364f7eb5f614947b48aa790585af70bba4029bfee7451e9f7b7494242 not found: ID does not exist" containerID="7bef49e364f7eb5f614947b48aa790585af70bba4029bfee7451e9f7b7494242"
Oct 30 20:29:25.970340 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:25.970285    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bef49e364f7eb5f614947b48aa790585af70bba4029bfee7451e9f7b7494242"} err="failed to get container status \"7bef49e364f7eb5f614947b48aa790585af70bba4029bfee7451e9f7b7494242\": rpc error: code = NotFound desc = could not find container \"7bef49e364f7eb5f614947b48aa790585af70bba4029bfee7451e9f7b7494242\": container with ID starting with 7bef49e364f7eb5f614947b48aa790585af70bba4029bfee7451e9f7b7494242 not found: ID does not exist"
Oct 30 20:29:26.125770 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:26.125732    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h"]
Oct 30 20:29:26.175590 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:26.175554    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-453277-6bcd89878b-xcj7h"]
Oct 30 20:29:27.398133 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:27.398100    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e12ca895-aa85-41db-89bf-d9912ba799f9" path="/var/lib/kubelet/pods/e12ca895-aa85-41db-89bf-d9912ba799f9/volumes"
Oct 30 20:29:35.231927 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.231888    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx" podUID="c9fa71b4-8a5b-40e6-be72-35583f6560f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:29:35.706587 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.706546    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-407632-d494w"]
Oct 30 20:29:35.707072 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.707056    5958 cpu_manager.go:414] "containerMap: removing container" podUID="e12ca895-aa85-41db-89bf-d9912ba799f9" containerName="maas-api"
Oct 30 20:29:35.707072 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.707072    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e12ca895-aa85-41db-89bf-d9912ba799f9" containerName="maas-api"
Oct 30 20:29:35.707175 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.707084    5958 cpu_manager.go:414] "containerMap: removing container" podUID="7578c703-e969-495b-98e9-70bfdf1f8413" containerName="payload-processing"
Oct 30 20:29:35.707175 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.707089    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7578c703-e969-495b-98e9-70bfdf1f8413" containerName="payload-processing"
Oct 30 20:29:35.707175 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.707097    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3224f5ec-0f50-454e-a07d-e16c0b9fe71b" containerName="payload-pre-processing"
Oct 30 20:29:35.707175 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.707102    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3224f5ec-0f50-454e-a07d-e16c0b9fe71b" containerName="payload-pre-processing"
Oct 30 20:29:35.707175 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.707116    5958 cpu_manager.go:414] "containerMap: removing container" podUID="60327e5a-6b34-4eff-bfaa-6af3715d0b78" containerName="payload-processing"
Oct 30 20:29:35.707175 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.707121    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60327e5a-6b34-4eff-bfaa-6af3715d0b78" containerName="payload-processing"
Oct 30 20:29:35.707175 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.707130    5958 cpu_manager.go:414] "containerMap: removing container" podUID="5e07ac50-c692-49c7-a2c3-dcbc364693ae" containerName="main"
Oct 30 20:29:35.707175 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.707135    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e07ac50-c692-49c7-a2c3-dcbc364693ae" containerName="main"
Oct 30 20:29:35.707175 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.707153    5958 cpu_manager.go:414] "containerMap: removing container" podUID="04b5051c-1bff-4550-b442-7f0b7622a271" containerName="istio-proxy"
Oct 30 20:29:35.707175 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.707157    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b5051c-1bff-4550-b442-7f0b7622a271" containerName="istio-proxy"
Oct 30 20:29:35.707175 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.707171    5958 cpu_manager.go:414] "containerMap: removing container" podUID="88103bf5-97bb-4a35-8906-51d55325f657" containerName="revoke-keys"
Oct 30 20:29:35.707175 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.707176    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88103bf5-97bb-4a35-8906-51d55325f657" containerName="revoke-keys"
Oct 30 20:29:35.707635 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.707243    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="88103bf5-97bb-4a35-8906-51d55325f657" containerName="revoke-keys"
Oct 30 20:29:35.707635 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.707255    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="e12ca895-aa85-41db-89bf-d9912ba799f9" containerName="maas-api"
Oct 30 20:29:35.707635 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.707260    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="7578c703-e969-495b-98e9-70bfdf1f8413" containerName="payload-processing"
Oct 30 20:29:35.707635 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.707265    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3224f5ec-0f50-454e-a07d-e16c0b9fe71b" containerName="payload-pre-processing"
Oct 30 20:29:35.707635 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.707272    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="60327e5a-6b34-4eff-bfaa-6af3715d0b78" containerName="payload-processing"
Oct 30 20:29:35.707635 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.707278    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e07ac50-c692-49c7-a2c3-dcbc364693ae" containerName="main"
Oct 30 20:29:35.711929 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.711906    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-407632-d494w"
Oct 30 20:29:35.816049 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.816009    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:29:35.816279 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.816108    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:29:35.816279 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.816150    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:29:35.816279 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.816213    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:29:35.818143 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.818115    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-407632-d494w"]
Oct 30 20:29:35.860963 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.860923    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db45d07e-8e54-4b01-abb2-31e7ad9fd864-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-407632-d494w\" (UID: \"db45d07e-8e54-4b01-abb2-31e7ad9fd864\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-407632-d494w"
Oct 30 20:29:35.861169 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.860988    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7dpt\" (UniqueName: \"kubernetes.io/projected/db45d07e-8e54-4b01-abb2-31e7ad9fd864-kube-api-access-b7dpt\") pod \"maas-api-revoke-keys-e2e-worker-w5-407632-d494w\" (UID: \"db45d07e-8e54-4b01-abb2-31e7ad9fd864\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-407632-d494w"
Oct 30 20:29:35.861169 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.861141    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db45d07e-8e54-4b01-abb2-31e7ad9fd864-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-407632-d494w\" (UID: \"db45d07e-8e54-4b01-abb2-31e7ad9fd864\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-407632-d494w"
Oct 30 20:29:35.962062 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.961981    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db45d07e-8e54-4b01-abb2-31e7ad9fd864-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-407632-d494w\" (UID: \"db45d07e-8e54-4b01-abb2-31e7ad9fd864\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-407632-d494w"
Oct 30 20:29:35.962062 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.962049    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7dpt\" (UniqueName: \"kubernetes.io/projected/db45d07e-8e54-4b01-abb2-31e7ad9fd864-kube-api-access-b7dpt\") pod \"maas-api-revoke-keys-e2e-worker-w5-407632-d494w\" (UID: \"db45d07e-8e54-4b01-abb2-31e7ad9fd864\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-407632-d494w"
Oct 30 20:29:35.962263 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.962106    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db45d07e-8e54-4b01-abb2-31e7ad9fd864-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-407632-d494w\" (UID: \"db45d07e-8e54-4b01-abb2-31e7ad9fd864\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-407632-d494w"
Oct 30 20:29:35.962686 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.962655    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db45d07e-8e54-4b01-abb2-31e7ad9fd864-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-407632-d494w\" (UID: \"db45d07e-8e54-4b01-abb2-31e7ad9fd864\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-407632-d494w"
Oct 30 20:29:35.964558 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:35.964539    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db45d07e-8e54-4b01-abb2-31e7ad9fd864-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-407632-d494w\" (UID: \"db45d07e-8e54-4b01-abb2-31e7ad9fd864\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-407632-d494w"
Oct 30 20:29:36.024032 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:36.023987    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7dpt\" (UniqueName: \"kubernetes.io/projected/db45d07e-8e54-4b01-abb2-31e7ad9fd864-kube-api-access-b7dpt\") pod \"maas-api-revoke-keys-e2e-worker-w5-407632-d494w\" (UID: \"db45d07e-8e54-4b01-abb2-31e7ad9fd864\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-407632-d494w"
Oct 30 20:29:36.025794 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:36.025776    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-407632-d494w"
Oct 30 20:29:36.344881 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:36.344851    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx"
Oct 30 20:29:36.399751 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:36.399719    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-407632-d494w"]
Oct 30 20:29:36.401777 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:29:36.401750    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb45d07e_8e54_4b01_abb2_31e7ad9fd864.slice/crio-d20618df49826c374f4d2e48dd5afdcbd2633af39094e4598296c443e28aa66c WatchSource:0}: Error finding container d20618df49826c374f4d2e48dd5afdcbd2633af39094e4598296c443e28aa66c: Status 404 returned error can't find the container with id d20618df49826c374f4d2e48dd5afdcbd2633af39094e4598296c443e28aa66c
Oct 30 20:29:36.467462 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:36.467368    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9fa71b4-8a5b-40e6-be72-35583f6560f5-kube-api-access-krlbr\" (UniqueName: \"kubernetes.io/projected/c9fa71b4-8a5b-40e6-be72-35583f6560f5-kube-api-access-krlbr\") pod \"c9fa71b4-8a5b-40e6-be72-35583f6560f5\" (UID: \"c9fa71b4-8a5b-40e6-be72-35583f6560f5\") "
Oct 30 20:29:36.467462 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:36.467423    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9fa71b4-8a5b-40e6-be72-35583f6560f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9fa71b4-8a5b-40e6-be72-35583f6560f5-plugins-config-volume\") pod \"c9fa71b4-8a5b-40e6-be72-35583f6560f5\" (UID: \"c9fa71b4-8a5b-40e6-be72-35583f6560f5\") "
Oct 30 20:29:36.467847 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:36.467816    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9fa71b4-8a5b-40e6-be72-35583f6560f5-plugins-config-volume" pod "c9fa71b4-8a5b-40e6-be72-35583f6560f5" (UID: "c9fa71b4-8a5b-40e6-be72-35583f6560f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:29:36.468159 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:36.468140    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9fa71b4-8a5b-40e6-be72-35583f6560f5-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:36.469731 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:36.469707    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9fa71b4-8a5b-40e6-be72-35583f6560f5-kube-api-access-krlbr" pod "c9fa71b4-8a5b-40e6-be72-35583f6560f5" (UID: "c9fa71b4-8a5b-40e6-be72-35583f6560f5"). InnerVolumeSpecName "kube-api-access-krlbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:36.568822 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:36.568795    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krlbr\" (UniqueName: \"kubernetes.io/projected/c9fa71b4-8a5b-40e6-be72-35583f6560f5-kube-api-access-krlbr\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:36.606676 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:29:36.606639    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:29:37.026658 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:37.026618    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-407632-d494w" event={"ID":"db45d07e-8e54-4b01-abb2-31e7ad9fd864","Type":"ContainerStarted","Data":"f4fc9a0583d527d85732fc08715d2fef1544c8f85e331b9bbcd40e96924e3a49"}
Oct 30 20:29:37.026895 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:37.026728    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-407632-d494w" event={"ID":"db45d07e-8e54-4b01-abb2-31e7ad9fd864","Type":"ContainerStarted","Data":"d20618df49826c374f4d2e48dd5afdcbd2633af39094e4598296c443e28aa66c"}
Oct 30 20:29:37.027888 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:37.027861    5958 generic.go:356] "Generic (PLEG): container finished" podID="c9fa71b4-8a5b-40e6-be72-35583f6560f5" containerID="b6fbd421df27db7bb75bd7f6ed2cade3c2e2879354336cdb04fb46e4fe4e00ab" exitCode=0
Oct 30 20:29:37.028000 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:37.027911    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx"
Oct 30 20:29:37.028000 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:37.027946    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx" event={"ID":"c9fa71b4-8a5b-40e6-be72-35583f6560f5","Type":"ContainerDied","Data":"b6fbd421df27db7bb75bd7f6ed2cade3c2e2879354336cdb04fb46e4fe4e00ab"}
Oct 30 20:29:37.028000 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:37.027996    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx" event={"ID":"c9fa71b4-8a5b-40e6-be72-35583f6560f5","Type":"ContainerDied","Data":"cca15862347dd09055215355748657fde10a84faf32fcf2c710c1a18495db70e"}
Oct 30 20:29:37.028102 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:37.028024    5958 scope.go:122] "RemoveContainer" containerID="b6fbd421df27db7bb75bd7f6ed2cade3c2e2879354336cdb04fb46e4fe4e00ab"
Oct 30 20:29:37.037740 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:37.037719    5958 scope.go:122] "RemoveContainer" containerID="b6fbd421df27db7bb75bd7f6ed2cade3c2e2879354336cdb04fb46e4fe4e00ab"
Oct 30 20:29:37.038009 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:29:37.037984    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6fbd421df27db7bb75bd7f6ed2cade3c2e2879354336cdb04fb46e4fe4e00ab\": container with ID starting with b6fbd421df27db7bb75bd7f6ed2cade3c2e2879354336cdb04fb46e4fe4e00ab not found: ID does not exist" containerID="b6fbd421df27db7bb75bd7f6ed2cade3c2e2879354336cdb04fb46e4fe4e00ab"
Oct 30 20:29:37.038095 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:37.038019    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6fbd421df27db7bb75bd7f6ed2cade3c2e2879354336cdb04fb46e4fe4e00ab"} err="failed to get container status \"b6fbd421df27db7bb75bd7f6ed2cade3c2e2879354336cdb04fb46e4fe4e00ab\": rpc error: code = NotFound desc = could not find container \"b6fbd421df27db7bb75bd7f6ed2cade3c2e2879354336cdb04fb46e4fe4e00ab\": container with ID starting with b6fbd421df27db7bb75bd7f6ed2cade3c2e2879354336cdb04fb46e4fe4e00ab not found: ID does not exist"
Oct 30 20:29:37.148148 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:37.148091    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-407632-d494w" podStartSLOduration=2.148076349 podStartE2EDuration="2.148076349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:37.146070933 +0000 UTC m=+2815.878286324" watchObservedRunningTime="2026-10-30 20:29:37.148076349 +0000 UTC m=+2815.880291739"
Oct 30 20:29:37.258550 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:37.258467    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx"]
Oct 30 20:29:37.262974 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:37.262947    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-453277-9dc9b578c-7dkgx"]
Oct 30 20:29:37.398540 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:37.398498    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9fa71b4-8a5b-40e6-be72-35583f6560f5" path="/var/lib/kubelet/pods/c9fa71b4-8a5b-40e6-be72-35583f6560f5/volumes"
Oct 30 20:29:38.035097 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:38.035059    5958 generic.go:356] "Generic (PLEG): container finished" podID="db45d07e-8e54-4b01-abb2-31e7ad9fd864" containerID="f4fc9a0583d527d85732fc08715d2fef1544c8f85e331b9bbcd40e96924e3a49" exitCode=0
Oct 30 20:29:38.035306 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:38.035151    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-407632-d494w" event={"ID":"db45d07e-8e54-4b01-abb2-31e7ad9fd864","Type":"ContainerDied","Data":"f4fc9a0583d527d85732fc08715d2fef1544c8f85e331b9bbcd40e96924e3a49"}
Oct 30 20:29:39.183860 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:39.183835    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-407632-d494w"
Oct 30 20:29:39.299981 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:39.299896    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db45d07e-8e54-4b01-abb2-31e7ad9fd864-kube-api-access-b7dpt\" (UniqueName: \"kubernetes.io/projected/db45d07e-8e54-4b01-abb2-31e7ad9fd864-kube-api-access-b7dpt\") pod \"db45d07e-8e54-4b01-abb2-31e7ad9fd864\" (UID: \"db45d07e-8e54-4b01-abb2-31e7ad9fd864\") "
Oct 30 20:29:39.300120 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:39.300027    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db45d07e-8e54-4b01-abb2-31e7ad9fd864-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db45d07e-8e54-4b01-abb2-31e7ad9fd864-maas-api-openshift-ca\") pod \"db45d07e-8e54-4b01-abb2-31e7ad9fd864\" (UID: \"db45d07e-8e54-4b01-abb2-31e7ad9fd864\") "
Oct 30 20:29:39.300120 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:39.300086    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db45d07e-8e54-4b01-abb2-31e7ad9fd864-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db45d07e-8e54-4b01-abb2-31e7ad9fd864-maas-api-xks-ca\") pod \"db45d07e-8e54-4b01-abb2-31e7ad9fd864\" (UID: \"db45d07e-8e54-4b01-abb2-31e7ad9fd864\") "
Oct 30 20:29:39.300444 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:39.300415    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db45d07e-8e54-4b01-abb2-31e7ad9fd864-maas-api-openshift-ca" pod "db45d07e-8e54-4b01-abb2-31e7ad9fd864" (UID: "db45d07e-8e54-4b01-abb2-31e7ad9fd864"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:29:39.304930 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:39.304896    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db45d07e-8e54-4b01-abb2-31e7ad9fd864-maas-api-xks-ca" pod "db45d07e-8e54-4b01-abb2-31e7ad9fd864" (UID: "db45d07e-8e54-4b01-abb2-31e7ad9fd864"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:29:39.305078 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:39.304936    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db45d07e-8e54-4b01-abb2-31e7ad9fd864-kube-api-access-b7dpt" pod "db45d07e-8e54-4b01-abb2-31e7ad9fd864" (UID: "db45d07e-8e54-4b01-abb2-31e7ad9fd864"). InnerVolumeSpecName "kube-api-access-b7dpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:39.401704 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:39.401669    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7dpt\" (UniqueName: \"kubernetes.io/projected/db45d07e-8e54-4b01-abb2-31e7ad9fd864-kube-api-access-b7dpt\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:39.401704 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:39.401697    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db45d07e-8e54-4b01-abb2-31e7ad9fd864-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:39.401704 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:39.401707    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db45d07e-8e54-4b01-abb2-31e7ad9fd864-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:40.047302 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:40.047274    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-407632-d494w"
Oct 30 20:29:40.047507 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:40.047306    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-407632-d494w" event={"ID":"db45d07e-8e54-4b01-abb2-31e7ad9fd864","Type":"ContainerDied","Data":"d20618df49826c374f4d2e48dd5afdcbd2633af39094e4598296c443e28aa66c"}
Oct 30 20:29:40.047507 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:40.047362    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d20618df49826c374f4d2e48dd5afdcbd2633af39094e4598296c443e28aa66c"
Oct 30 20:29:47.605775 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:47.605735    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-14c846-jwww4"]
Oct 30 20:29:47.606269 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:47.606218    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c9fa71b4-8a5b-40e6-be72-35583f6560f5" containerName="payload-pre-processing"
Oct 30 20:29:47.606269 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:47.606229    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9fa71b4-8a5b-40e6-be72-35583f6560f5" containerName="payload-pre-processing"
Oct 30 20:29:47.606269 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:47.606248    5958 cpu_manager.go:414] "containerMap: removing container" podUID="db45d07e-8e54-4b01-abb2-31e7ad9fd864" containerName="revoke-keys"
Oct 30 20:29:47.606269 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:47.606253    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db45d07e-8e54-4b01-abb2-31e7ad9fd864" containerName="revoke-keys"
Oct 30 20:29:47.606589 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:47.606322    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="db45d07e-8e54-4b01-abb2-31e7ad9fd864" containerName="revoke-keys"
Oct 30 20:29:47.606589 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:47.606338    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9fa71b4-8a5b-40e6-be72-35583f6560f5" containerName="payload-pre-processing"
Oct 30 20:29:47.611115 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:47.611098    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-14c846-jwww4"
Oct 30 20:29:47.637384 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:47.637356    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-14c846-jwww4"]
Oct 30 20:29:47.665037 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:47.665007    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:29:47.665173 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:47.665156    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:29:47.665352 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:47.665339    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:29:47.665441 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:47.665424    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:29:47.798352 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:47.798307    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf455d6d-8843-4c51-bf9d-07135b9577e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-14c846-jwww4\" (UID: \"bf455d6d-8843-4c51-bf9d-07135b9577e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-14c846-jwww4"
Oct 30 20:29:47.798564 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:47.798488    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5vhn\" (UniqueName: \"kubernetes.io/projected/bf455d6d-8843-4c51-bf9d-07135b9577e8-kube-api-access-j5vhn\") pod \"maas-api-revoke-keys-e2e-inf-a-14c846-jwww4\" (UID: \"bf455d6d-8843-4c51-bf9d-07135b9577e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-14c846-jwww4"
Oct 30 20:29:47.798564 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:47.798553    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf455d6d-8843-4c51-bf9d-07135b9577e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-14c846-jwww4\" (UID: \"bf455d6d-8843-4c51-bf9d-07135b9577e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-14c846-jwww4"
Oct 30 20:29:47.899702 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:47.899610    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5vhn\" (UniqueName: \"kubernetes.io/projected/bf455d6d-8843-4c51-bf9d-07135b9577e8-kube-api-access-j5vhn\") pod \"maas-api-revoke-keys-e2e-inf-a-14c846-jwww4\" (UID: \"bf455d6d-8843-4c51-bf9d-07135b9577e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-14c846-jwww4"
Oct 30 20:29:47.899702 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:47.899667    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf455d6d-8843-4c51-bf9d-07135b9577e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-14c846-jwww4\" (UID: \"bf455d6d-8843-4c51-bf9d-07135b9577e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-14c846-jwww4"
Oct 30 20:29:47.899943 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:47.899703    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf455d6d-8843-4c51-bf9d-07135b9577e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-14c846-jwww4\" (UID: \"bf455d6d-8843-4c51-bf9d-07135b9577e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-14c846-jwww4"
Oct 30 20:29:47.900323 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:47.900300    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf455d6d-8843-4c51-bf9d-07135b9577e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-14c846-jwww4\" (UID: \"bf455d6d-8843-4c51-bf9d-07135b9577e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-14c846-jwww4"
Oct 30 20:29:47.902250 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:47.902226    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf455d6d-8843-4c51-bf9d-07135b9577e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-14c846-jwww4\" (UID: \"bf455d6d-8843-4c51-bf9d-07135b9577e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-14c846-jwww4"
Oct 30 20:29:47.964710 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:47.964679    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5vhn\" (UniqueName: \"kubernetes.io/projected/bf455d6d-8843-4c51-bf9d-07135b9577e8-kube-api-access-j5vhn\") pod \"maas-api-revoke-keys-e2e-inf-a-14c846-jwww4\" (UID: \"bf455d6d-8843-4c51-bf9d-07135b9577e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-14c846-jwww4"
Oct 30 20:29:48.223878 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:48.223807    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-14c846-jwww4"
Oct 30 20:29:48.597826 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:48.597802    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-14c846-jwww4"]
Oct 30 20:29:48.599966 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:29:48.599937    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf455d6d_8843_4c51_bf9d_07135b9577e8.slice/crio-c4afa60fba6b3dd3cd5d75488618b61369a8272f434364bfff5b00e444614d86 WatchSource:0}: Error finding container c4afa60fba6b3dd3cd5d75488618b61369a8272f434364bfff5b00e444614d86: Status 404 returned error can't find the container with id c4afa60fba6b3dd3cd5d75488618b61369a8272f434364bfff5b00e444614d86
Oct 30 20:29:49.097848 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:49.097809    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-14c846-jwww4" event={"ID":"bf455d6d-8843-4c51-bf9d-07135b9577e8","Type":"ContainerStarted","Data":"de5f3742666b270344deb90f058f902c00545ee1b4121d0dd60f02415326aa4e"}
Oct 30 20:29:49.098268 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:49.097874    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-14c846-jwww4" event={"ID":"bf455d6d-8843-4c51-bf9d-07135b9577e8","Type":"ContainerStarted","Data":"c4afa60fba6b3dd3cd5d75488618b61369a8272f434364bfff5b00e444614d86"}
Oct 30 20:29:49.217936 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:49.217877    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-14c846-jwww4" podStartSLOduration=2.217861959 podStartE2EDuration="2.217861959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:49.215756343 +0000 UTC m=+2827.947971736" watchObservedRunningTime="2026-10-30 20:29:49.217861959 +0000 UTC m=+2827.950077349"
Oct 30 20:29:50.105263 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:50.105225    5958 generic.go:356] "Generic (PLEG): container finished" podID="bf455d6d-8843-4c51-bf9d-07135b9577e8" containerID="de5f3742666b270344deb90f058f902c00545ee1b4121d0dd60f02415326aa4e" exitCode=0
Oct 30 20:29:50.105668 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:50.105308    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-14c846-jwww4" event={"ID":"bf455d6d-8843-4c51-bf9d-07135b9577e8","Type":"ContainerDied","Data":"de5f3742666b270344deb90f058f902c00545ee1b4121d0dd60f02415326aa4e"}
Oct 30 20:29:51.251303 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:51.251277    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-14c846-jwww4"
Oct 30 20:29:51.330247 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:51.330218    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf455d6d-8843-4c51-bf9d-07135b9577e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf455d6d-8843-4c51-bf9d-07135b9577e8-maas-api-xks-ca\") pod \"bf455d6d-8843-4c51-bf9d-07135b9577e8\" (UID: \"bf455d6d-8843-4c51-bf9d-07135b9577e8\") "
Oct 30 20:29:51.330397 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:51.330276    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf455d6d-8843-4c51-bf9d-07135b9577e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf455d6d-8843-4c51-bf9d-07135b9577e8-maas-api-openshift-ca\") pod \"bf455d6d-8843-4c51-bf9d-07135b9577e8\" (UID: \"bf455d6d-8843-4c51-bf9d-07135b9577e8\") "
Oct 30 20:29:51.330491 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:51.330409    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf455d6d-8843-4c51-bf9d-07135b9577e8-kube-api-access-j5vhn\" (UniqueName: \"kubernetes.io/projected/bf455d6d-8843-4c51-bf9d-07135b9577e8-kube-api-access-j5vhn\") pod \"bf455d6d-8843-4c51-bf9d-07135b9577e8\" (UID: \"bf455d6d-8843-4c51-bf9d-07135b9577e8\") "
Oct 30 20:29:51.330579 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:51.330547    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf455d6d-8843-4c51-bf9d-07135b9577e8-maas-api-openshift-ca" pod "bf455d6d-8843-4c51-bf9d-07135b9577e8" (UID: "bf455d6d-8843-4c51-bf9d-07135b9577e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:29:51.330696 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:51.330683    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf455d6d-8843-4c51-bf9d-07135b9577e8-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:51.332387 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:51.332360    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf455d6d-8843-4c51-bf9d-07135b9577e8-maas-api-xks-ca" pod "bf455d6d-8843-4c51-bf9d-07135b9577e8" (UID: "bf455d6d-8843-4c51-bf9d-07135b9577e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:29:51.332518 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:51.332393    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf455d6d-8843-4c51-bf9d-07135b9577e8-kube-api-access-j5vhn" pod "bf455d6d-8843-4c51-bf9d-07135b9577e8" (UID: "bf455d6d-8843-4c51-bf9d-07135b9577e8"). InnerVolumeSpecName "kube-api-access-j5vhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:29:51.431789 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:51.431759    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5vhn\" (UniqueName: \"kubernetes.io/projected/bf455d6d-8843-4c51-bf9d-07135b9577e8-kube-api-access-j5vhn\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:51.431789 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:51.431786    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf455d6d-8843-4c51-bf9d-07135b9577e8-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:29:51.433278 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:51.433252    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82"]
Oct 30 20:29:51.543504 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:51.543475    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm"]
Oct 30 20:29:52.116846 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:52.116815    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-14c846-jwww4"
Oct 30 20:29:52.117009 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:52.116819    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-14c846-jwww4" event={"ID":"bf455d6d-8843-4c51-bf9d-07135b9577e8","Type":"ContainerDied","Data":"c4afa60fba6b3dd3cd5d75488618b61369a8272f434364bfff5b00e444614d86"}
Oct 30 20:29:52.117009 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:52.116960    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4afa60fba6b3dd3cd5d75488618b61369a8272f434364bfff5b00e444614d86"
Oct 30 20:29:56.434203 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:56.434155    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82" podUID="4bfb13d3-e216-41f2-996d-11541708e890" containerName="payload-processing" containerID="cri-o://29327240b0ad52a5e4f5030c7582e87efcc50f187c1db4488f2252c76b738775" gracePeriod=25
Oct 30 20:29:56.544123 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:56.544059    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm" podUID="f17664d0-97ab-432b-8e09-dbf95dc52b15" containerName="payload-pre-processing" containerID="cri-o://83cc183fc372d25dea8c7a9ea816929e623b0981abeeb81907b2005b201ea67d" gracePeriod=25
Oct 30 20:29:59.248610 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:59.248571    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6"]
Oct 30 20:29:59.359605 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:29:59.359571    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986"]
Oct 30 20:30:00.169487 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.169425    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz"]
Oct 30 20:30:00.170080 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.170058    5958 cpu_manager.go:414] "containerMap: removing container" podUID="bf455d6d-8843-4c51-bf9d-07135b9577e8" containerName="revoke-keys"
Oct 30 20:30:00.170204 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.170082    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf455d6d-8843-4c51-bf9d-07135b9577e8" containerName="revoke-keys"
Oct 30 20:30:00.170204 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.170189    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf455d6d-8843-4c51-bf9d-07135b9577e8" containerName="revoke-keys"
Oct 30 20:30:00.173733 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.173711    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx"]
Oct 30 20:30:00.173886 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.173864    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz"
Oct 30 20:30:00.177500 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.177479    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx"
Oct 30 20:30:00.179495 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.179472    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw"]
Oct 30 20:30:00.183719 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.183698    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw"
Oct 30 20:30:00.226729 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.226705    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:30:00.227965 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.227943    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:30:00.228079 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.228057    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:30:00.229470 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.229434    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz6fx\" (UniqueName: \"kubernetes.io/projected/7beeda33-6b1f-40f1-a473-a5e84e2414a6-kube-api-access-wz6fx\") pod \"maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx\" (UID: \"7beeda33-6b1f-40f1-a473-a5e84e2414a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx"
Oct 30 20:30:00.229541 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.229483    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32b872e0-6cb5-48ef-9260-5a41099ac864-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889870-r8gbz\" (UID: \"32b872e0-6cb5-48ef-9260-5a41099ac864\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz"
Oct 30 20:30:00.229590 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.229573    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz879\" (UniqueName: \"kubernetes.io/projected/32b872e0-6cb5-48ef-9260-5a41099ac864-kube-api-access-bz879\") pod \"maas-api-key-cleanup-29889870-r8gbz\" (UID: \"32b872e0-6cb5-48ef-9260-5a41099ac864\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz"
Oct 30 20:30:00.229714 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.229699    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsqpl\" (UniqueName: \"kubernetes.io/projected/8e5a7c3a-bb1a-425e-b0db-934e55ac702f-kube-api-access-lsqpl\") pod \"maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw\" (UID: \"8e5a7c3a-bb1a-425e-b0db-934e55ac702f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw"
Oct 30 20:30:00.229780 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.229765    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7beeda33-6b1f-40f1-a473-a5e84e2414a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx\" (UID: \"7beeda33-6b1f-40f1-a473-a5e84e2414a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx"
Oct 30 20:30:00.229887 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.229865    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5a7c3a-bb1a-425e-b0db-934e55ac702f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw\" (UID: \"8e5a7c3a-bb1a-425e-b0db-934e55ac702f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw"
Oct 30 20:30:00.265830 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.265801    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz"]
Oct 30 20:30:00.272644 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.272619    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx"]
Oct 30 20:30:00.302427 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.302391    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm" podUID="f17664d0-97ab-432b-8e09-dbf95dc52b15" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:30:00.302616 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.302503    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82" podUID="4bfb13d3-e216-41f2-996d-11541708e890" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:30:00.331275 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.331231    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw"]
Oct 30 20:30:00.331438 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.331375    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsqpl\" (UniqueName: \"kubernetes.io/projected/8e5a7c3a-bb1a-425e-b0db-934e55ac702f-kube-api-access-lsqpl\") pod \"maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw\" (UID: \"8e5a7c3a-bb1a-425e-b0db-934e55ac702f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw"
Oct 30 20:30:00.331518 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.331506    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7beeda33-6b1f-40f1-a473-a5e84e2414a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx\" (UID: \"7beeda33-6b1f-40f1-a473-a5e84e2414a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx"
Oct 30 20:30:00.331563 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.331536    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5a7c3a-bb1a-425e-b0db-934e55ac702f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw\" (UID: \"8e5a7c3a-bb1a-425e-b0db-934e55ac702f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw"
Oct 30 20:30:00.331621 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.331567    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz6fx\" (UniqueName: \"kubernetes.io/projected/7beeda33-6b1f-40f1-a473-a5e84e2414a6-kube-api-access-wz6fx\") pod \"maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx\" (UID: \"7beeda33-6b1f-40f1-a473-a5e84e2414a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx"
Oct 30 20:30:00.331732 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.331711    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32b872e0-6cb5-48ef-9260-5a41099ac864-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889870-r8gbz\" (UID: \"32b872e0-6cb5-48ef-9260-5a41099ac864\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz"
Oct 30 20:30:00.331817 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.331772    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz879\" (UniqueName: \"kubernetes.io/projected/32b872e0-6cb5-48ef-9260-5a41099ac864-kube-api-access-bz879\") pod \"maas-api-key-cleanup-29889870-r8gbz\" (UID: \"32b872e0-6cb5-48ef-9260-5a41099ac864\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz"
Oct 30 20:30:00.332348 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.332322    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7beeda33-6b1f-40f1-a473-a5e84e2414a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx\" (UID: \"7beeda33-6b1f-40f1-a473-a5e84e2414a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx"
Oct 30 20:30:00.332550 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.332528    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32b872e0-6cb5-48ef-9260-5a41099ac864-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889870-r8gbz\" (UID: \"32b872e0-6cb5-48ef-9260-5a41099ac864\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz"
Oct 30 20:30:00.332609 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.332528    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5a7c3a-bb1a-425e-b0db-934e55ac702f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw\" (UID: \"8e5a7c3a-bb1a-425e-b0db-934e55ac702f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw"
Oct 30 20:30:00.392889 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.392855    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsqpl\" (UniqueName: \"kubernetes.io/projected/8e5a7c3a-bb1a-425e-b0db-934e55ac702f-kube-api-access-lsqpl\") pod \"maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw\" (UID: \"8e5a7c3a-bb1a-425e-b0db-934e55ac702f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw"
Oct 30 20:30:00.393032 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.392940    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz6fx\" (UniqueName: \"kubernetes.io/projected/7beeda33-6b1f-40f1-a473-a5e84e2414a6-kube-api-access-wz6fx\") pod \"maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx\" (UID: \"7beeda33-6b1f-40f1-a473-a5e84e2414a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx"
Oct 30 20:30:00.393032 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.392944    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz879\" (UniqueName: \"kubernetes.io/projected/32b872e0-6cb5-48ef-9260-5a41099ac864-kube-api-access-bz879\") pod \"maas-api-key-cleanup-29889870-r8gbz\" (UID: \"32b872e0-6cb5-48ef-9260-5a41099ac864\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz"
Oct 30 20:30:00.488779 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.488714    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz"
Oct 30 20:30:00.495808 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.495780    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx"
Oct 30 20:30:00.502781 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.502756    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw"
Oct 30 20:30:00.871524 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.871496    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz"]
Oct 30 20:30:00.875537 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:30:00.875510    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b872e0_6cb5_48ef_9260_5a41099ac864.slice/crio-78881d594650d85f58f2911f252b41e6467abe79e4eedf9d8883ab05b7ed0a3c WatchSource:0}: Error finding container 78881d594650d85f58f2911f252b41e6467abe79e4eedf9d8883ab05b7ed0a3c: Status 404 returned error can't find the container with id 78881d594650d85f58f2911f252b41e6467abe79e4eedf9d8883ab05b7ed0a3c
Oct 30 20:30:00.925173 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.925135    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw"]
Oct 30 20:30:00.926511 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:30:00.926473    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5a7c3a_bb1a_425e_b0db_934e55ac702f.slice/crio-7a9f5dda9b076416e16ccdb4b80d577dca2fad019b186489eeefdd811e40fde5 WatchSource:0}: Error finding container 7a9f5dda9b076416e16ccdb4b80d577dca2fad019b186489eeefdd811e40fde5: Status 404 returned error can't find the container with id 7a9f5dda9b076416e16ccdb4b80d577dca2fad019b186489eeefdd811e40fde5
Oct 30 20:30:00.926958 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:00.926852    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx"]
Oct 30 20:30:00.927687 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:30:00.927654    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7beeda33_6b1f_40f1_a473_a5e84e2414a6.slice/crio-9397c2e23fb7480f8443ccfbc2acbf83dae1e417a7af8b4503209596fb68723a WatchSource:0}: Error finding container 9397c2e23fb7480f8443ccfbc2acbf83dae1e417a7af8b4503209596fb68723a: Status 404 returned error can't find the container with id 9397c2e23fb7480f8443ccfbc2acbf83dae1e417a7af8b4503209596fb68723a
Oct 30 20:30:01.169999 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:01.169967    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw" event={"ID":"8e5a7c3a-bb1a-425e-b0db-934e55ac702f","Type":"ContainerStarted","Data":"7a9f5dda9b076416e16ccdb4b80d577dca2fad019b186489eeefdd811e40fde5"}
Oct 30 20:30:01.171139 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:01.171112    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx" event={"ID":"7beeda33-6b1f-40f1-a473-a5e84e2414a6","Type":"ContainerStarted","Data":"9397c2e23fb7480f8443ccfbc2acbf83dae1e417a7af8b4503209596fb68723a"}
Oct 30 20:30:01.172527 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:01.172507    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz" event={"ID":"32b872e0-6cb5-48ef-9260-5a41099ac864","Type":"ContainerStarted","Data":"6c2191ccfa18d8a914bc09d773fcebb0b6050d0f66181908d3c97f63f63ac4ca"}
Oct 30 20:30:01.172609 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:01.172543    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz" event={"ID":"32b872e0-6cb5-48ef-9260-5a41099ac864","Type":"ContainerStarted","Data":"78881d594650d85f58f2911f252b41e6467abe79e4eedf9d8883ab05b7ed0a3c"}
Oct 30 20:30:01.273012 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:01.272982    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55a47471-2p599"]
Oct 30 20:30:01.276853 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:01.276830    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a47471-2p599"
Oct 30 20:30:01.288898 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:01.288853    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz" podStartSLOduration=1.2888408550000001 podStartE2EDuration="1.288840855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:01.287699717 +0000 UTC m=+2840.019915108" watchObservedRunningTime="2026-10-30 20:30:01.288840855 +0000 UTC m=+2840.021056244"
Oct 30 20:30:01.343393 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:01.343363    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a190a0-8198-4463-b199-dd7b7515425c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55a47471-2p599\" (UID: \"a6a190a0-8198-4463-b199-dd7b7515425c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a47471-2p599"
Oct 30 20:30:01.343542 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:01.343399    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a190a0-8198-4463-b199-dd7b7515425c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55a47471-2p599\" (UID: \"a6a190a0-8198-4463-b199-dd7b7515425c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a47471-2p599"
Oct 30 20:30:01.343542 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:01.343424    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gppb\" (UniqueName: \"kubernetes.io/projected/a6a190a0-8198-4463-b199-dd7b7515425c-kube-api-access-4gppb\") pod \"maas-api-revoke-sub-55a47471-2p599\" (UID: \"a6a190a0-8198-4463-b199-dd7b7515425c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a47471-2p599"
Oct 30 20:30:01.358782 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:01.358731    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55a47471-2p599"]
Oct 30 20:30:01.380609 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:01.380587    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:30:01.444130 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:01.444075    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a190a0-8198-4463-b199-dd7b7515425c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55a47471-2p599\" (UID: \"a6a190a0-8198-4463-b199-dd7b7515425c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a47471-2p599"
Oct 30 20:30:01.444130 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:01.444104    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a190a0-8198-4463-b199-dd7b7515425c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55a47471-2p599\" (UID: \"a6a190a0-8198-4463-b199-dd7b7515425c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a47471-2p599"
Oct 30 20:30:01.444130 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:01.444124    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gppb\" (UniqueName: \"kubernetes.io/projected/a6a190a0-8198-4463-b199-dd7b7515425c-kube-api-access-4gppb\") pod \"maas-api-revoke-sub-55a47471-2p599\" (UID: \"a6a190a0-8198-4463-b199-dd7b7515425c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a47471-2p599"
Oct 30 20:30:01.444685 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:01.444667    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a190a0-8198-4463-b199-dd7b7515425c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55a47471-2p599\" (UID: \"a6a190a0-8198-4463-b199-dd7b7515425c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a47471-2p599"
Oct 30 20:30:01.446574 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:01.446551    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a190a0-8198-4463-b199-dd7b7515425c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55a47471-2p599\" (UID: \"a6a190a0-8198-4463-b199-dd7b7515425c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a47471-2p599"
Oct 30 20:30:01.505301 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:01.505278    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gppb\" (UniqueName: \"kubernetes.io/projected/a6a190a0-8198-4463-b199-dd7b7515425c-kube-api-access-4gppb\") pod \"maas-api-revoke-sub-55a47471-2p599\" (UID: \"a6a190a0-8198-4463-b199-dd7b7515425c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a47471-2p599"
Oct 30 20:30:01.590914 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:01.590892    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a47471-2p599"
Oct 30 20:30:01.962677 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:01.962654    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55a47471-2p599"]
Oct 30 20:30:01.964499 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:30:01.964478    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a190a0_8198_4463_b199_dd7b7515425c.slice/crio-d099ff06f337024505f4f087a00ce24ec6d544709ee19d0e20024e82ca15143f WatchSource:0}: Error finding container d099ff06f337024505f4f087a00ce24ec6d544709ee19d0e20024e82ca15143f: Status 404 returned error can't find the container with id d099ff06f337024505f4f087a00ce24ec6d544709ee19d0e20024e82ca15143f
Oct 30 20:30:02.179842 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:02.179772    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a47471-2p599" event={"ID":"a6a190a0-8198-4463-b199-dd7b7515425c","Type":"ContainerStarted","Data":"d099ff06f337024505f4f087a00ce24ec6d544709ee19d0e20024e82ca15143f"}
Oct 30 20:30:02.181814 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:02.181790    5958 generic.go:356] "Generic (PLEG): container finished" podID="7beeda33-6b1f-40f1-a473-a5e84e2414a6" containerID="debdc611f739c30fcc104350da1f8a7b56903c35c6d38befeba403adf8ca34db" exitCode=60
Oct 30 20:30:02.181948 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:02.181864    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx" event={"ID":"7beeda33-6b1f-40f1-a473-a5e84e2414a6","Type":"ContainerDied","Data":"debdc611f739c30fcc104350da1f8a7b56903c35c6d38befeba403adf8ca34db"}
Oct 30 20:30:02.182105 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:02.182087    5958 scope.go:122] "RemoveContainer" containerID="debdc611f739c30fcc104350da1f8a7b56903c35c6d38befeba403adf8ca34db"
Oct 30 20:30:02.183853 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:02.183830    5958 generic.go:356] "Generic (PLEG): container finished" podID="32b872e0-6cb5-48ef-9260-5a41099ac864" containerID="6c2191ccfa18d8a914bc09d773fcebb0b6050d0f66181908d3c97f63f63ac4ca" exitCode=60
Oct 30 20:30:02.184007 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:02.183978    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz" event={"ID":"32b872e0-6cb5-48ef-9260-5a41099ac864","Type":"ContainerDied","Data":"6c2191ccfa18d8a914bc09d773fcebb0b6050d0f66181908d3c97f63f63ac4ca"}
Oct 30 20:30:02.184152 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:02.184131    5958 scope.go:122] "RemoveContainer" containerID="6c2191ccfa18d8a914bc09d773fcebb0b6050d0f66181908d3c97f63f63ac4ca"
Oct 30 20:30:02.186742 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:02.186702    5958 generic.go:356] "Generic (PLEG): container finished" podID="8e5a7c3a-bb1a-425e-b0db-934e55ac702f" containerID="8d4ac2674e89655c8522afce2391bdc4d0ecc3c7807ba9e112b76668d598e405" exitCode=60
Oct 30 20:30:02.186742 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:02.186729    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw" event={"ID":"8e5a7c3a-bb1a-425e-b0db-934e55ac702f","Type":"ContainerDied","Data":"8d4ac2674e89655c8522afce2391bdc4d0ecc3c7807ba9e112b76668d598e405"}
Oct 30 20:30:02.186999 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:02.186978    5958 scope.go:122] "RemoveContainer" containerID="8d4ac2674e89655c8522afce2391bdc4d0ecc3c7807ba9e112b76668d598e405"
Oct 30 20:30:03.193823 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:03.193754    5958 generic.go:356] "Generic (PLEG): container finished" podID="32b872e0-6cb5-48ef-9260-5a41099ac864" containerID="71eb3c49f0103637d7f9b8a769910d5c31b8b2c9b06bff33105f3de57a11dd54" exitCode=60
Oct 30 20:30:03.194197 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:03.193835    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz" event={"ID":"32b872e0-6cb5-48ef-9260-5a41099ac864","Type":"ContainerDied","Data":"71eb3c49f0103637d7f9b8a769910d5c31b8b2c9b06bff33105f3de57a11dd54"}
Oct 30 20:30:03.194197 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:03.193890    5958 scope.go:122] "RemoveContainer" containerID="6c2191ccfa18d8a914bc09d773fcebb0b6050d0f66181908d3c97f63f63ac4ca"
Oct 30 20:30:03.194197 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:03.194056    5958 scope.go:122] "RemoveContainer" containerID="71eb3c49f0103637d7f9b8a769910d5c31b8b2c9b06bff33105f3de57a11dd54"
Oct 30 20:30:03.194356 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:30:03.194309    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889870-r8gbz_odh-ai-gateway-infra(32b872e0-6cb5-48ef-9260-5a41099ac864)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz" podUID="32b872e0-6cb5-48ef-9260-5a41099ac864"
Oct 30 20:30:03.195966 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:03.195947    5958 generic.go:356] "Generic (PLEG): container finished" podID="8e5a7c3a-bb1a-425e-b0db-934e55ac702f" containerID="3a57a556791202b60771fb1abb64fac3351a68adcd02ba3a287e05f852fb1c49" exitCode=60
Oct 30 20:30:03.196068 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:03.196037    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw" event={"ID":"8e5a7c3a-bb1a-425e-b0db-934e55ac702f","Type":"ContainerDied","Data":"3a57a556791202b60771fb1abb64fac3351a68adcd02ba3a287e05f852fb1c49"}
Oct 30 20:30:03.196303 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:03.196269    5958 scope.go:122] "RemoveContainer" containerID="3a57a556791202b60771fb1abb64fac3351a68adcd02ba3a287e05f852fb1c49"
Oct 30 20:30:03.196690 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:30:03.196528    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw_odh-ai-gateway-infra(8e5a7c3a-bb1a-425e-b0db-934e55ac702f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw" podUID="8e5a7c3a-bb1a-425e-b0db-934e55ac702f"
Oct 30 20:30:03.197814 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:03.197793    5958 generic.go:356] "Generic (PLEG): container finished" podID="a6a190a0-8198-4463-b199-dd7b7515425c" containerID="26f209e87ddd7fc2784fea80f9c3f18bf1eed2316452c56ac24518bd41edc17e" exitCode=0
Oct 30 20:30:03.197913 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:03.197869    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a47471-2p599" event={"ID":"a6a190a0-8198-4463-b199-dd7b7515425c","Type":"ContainerDied","Data":"26f209e87ddd7fc2784fea80f9c3f18bf1eed2316452c56ac24518bd41edc17e"}
Oct 30 20:30:03.199636 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:03.199616    5958 generic.go:356] "Generic (PLEG): container finished" podID="7beeda33-6b1f-40f1-a473-a5e84e2414a6" containerID="cd287b9ebe7870a19788c1e51312be7a5e8e8e64d25d30a9ed1caa05a33f8319" exitCode=60
Oct 30 20:30:03.199713 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:03.199651    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx" event={"ID":"7beeda33-6b1f-40f1-a473-a5e84e2414a6","Type":"ContainerDied","Data":"cd287b9ebe7870a19788c1e51312be7a5e8e8e64d25d30a9ed1caa05a33f8319"}
Oct 30 20:30:03.199856 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:03.199843    5958 scope.go:122] "RemoveContainer" containerID="cd287b9ebe7870a19788c1e51312be7a5e8e8e64d25d30a9ed1caa05a33f8319"
Oct 30 20:30:03.200031 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:30:03.200015    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx_odh-ai-gateway-infra(7beeda33-6b1f-40f1-a473-a5e84e2414a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx" podUID="7beeda33-6b1f-40f1-a473-a5e84e2414a6"
Oct 30 20:30:03.209199 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:03.209182    5958 scope.go:122] "RemoveContainer" containerID="8d4ac2674e89655c8522afce2391bdc4d0ecc3c7807ba9e112b76668d598e405"
Oct 30 20:30:03.221004 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:03.220987    5958 scope.go:122] "RemoveContainer" containerID="debdc611f739c30fcc104350da1f8a7b56903c35c6d38befeba403adf8ca34db"
Oct 30 20:30:04.206650 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:04.206607    5958 scope.go:122] "RemoveContainer" containerID="cd287b9ebe7870a19788c1e51312be7a5e8e8e64d25d30a9ed1caa05a33f8319"
Oct 30 20:30:04.207091 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:30:04.206898    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx_odh-ai-gateway-infra(7beeda33-6b1f-40f1-a473-a5e84e2414a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx" podUID="7beeda33-6b1f-40f1-a473-a5e84e2414a6"
Oct 30 20:30:04.208390 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:04.208362    5958 scope.go:122] "RemoveContainer" containerID="71eb3c49f0103637d7f9b8a769910d5c31b8b2c9b06bff33105f3de57a11dd54"
Oct 30 20:30:04.208647 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:30:04.208627    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889870-r8gbz_odh-ai-gateway-infra(32b872e0-6cb5-48ef-9260-5a41099ac864)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz" podUID="32b872e0-6cb5-48ef-9260-5a41099ac864"
Oct 30 20:30:04.209881 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:04.209862    5958 scope.go:122] "RemoveContainer" containerID="3a57a556791202b60771fb1abb64fac3351a68adcd02ba3a287e05f852fb1c49"
Oct 30 20:30:04.210347 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:30:04.210125    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw_odh-ai-gateway-infra(8e5a7c3a-bb1a-425e-b0db-934e55ac702f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw" podUID="8e5a7c3a-bb1a-425e-b0db-934e55ac702f"
Oct 30 20:30:04.248904 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:04.248851    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6" podUID="2773ad9c-dde9-409c-937d-2a85268696f1" containerName="payload-processing" containerID="cri-o://824047327c083de6da5d33c09ff11b0f9fa1aba65fb5b65a16b059097e4c4ba8" gracePeriod=25
Oct 30 20:30:04.351929 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:04.351900    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a47471-2p599"
Oct 30 20:30:04.360305 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:04.360274    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986" podUID="ee6329a5-2f0b-47c7-84ec-25d1c72ba396" containerName="payload-pre-processing" containerID="cri-o://5e9aab54d9a95dd3a734df96406946af48a01179ad8629fa21fab3ebc656e95d" gracePeriod=25
Oct 30 20:30:04.475573 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:04.475495    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6a190a0-8198-4463-b199-dd7b7515425c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a190a0-8198-4463-b199-dd7b7515425c-maas-api-xks-ca\") pod \"a6a190a0-8198-4463-b199-dd7b7515425c\" (UID: \"a6a190a0-8198-4463-b199-dd7b7515425c\") "
Oct 30 20:30:04.475573 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:04.475532    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6a190a0-8198-4463-b199-dd7b7515425c-kube-api-access-4gppb\" (UniqueName: \"kubernetes.io/projected/a6a190a0-8198-4463-b199-dd7b7515425c-kube-api-access-4gppb\") pod \"a6a190a0-8198-4463-b199-dd7b7515425c\" (UID: \"a6a190a0-8198-4463-b199-dd7b7515425c\") "
Oct 30 20:30:04.475774 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:04.475591    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6a190a0-8198-4463-b199-dd7b7515425c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a190a0-8198-4463-b199-dd7b7515425c-maas-api-openshift-ca\") pod \"a6a190a0-8198-4463-b199-dd7b7515425c\" (UID: \"a6a190a0-8198-4463-b199-dd7b7515425c\") "
Oct 30 20:30:04.476057 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:04.476026    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6a190a0-8198-4463-b199-dd7b7515425c-maas-api-openshift-ca" pod "a6a190a0-8198-4463-b199-dd7b7515425c" (UID: "a6a190a0-8198-4463-b199-dd7b7515425c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:04.477823 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:04.477801    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6a190a0-8198-4463-b199-dd7b7515425c-maas-api-xks-ca" pod "a6a190a0-8198-4463-b199-dd7b7515425c" (UID: "a6a190a0-8198-4463-b199-dd7b7515425c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:30:04.477823 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:04.477811    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6a190a0-8198-4463-b199-dd7b7515425c-kube-api-access-4gppb" pod "a6a190a0-8198-4463-b199-dd7b7515425c" (UID: "a6a190a0-8198-4463-b199-dd7b7515425c"). InnerVolumeSpecName "kube-api-access-4gppb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:04.576637 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:04.576607    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6a190a0-8198-4463-b199-dd7b7515425c-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:04.576637 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:04.576634    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gppb\" (UniqueName: \"kubernetes.io/projected/a6a190a0-8198-4463-b199-dd7b7515425c-kube-api-access-4gppb\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:04.576797 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:04.576644    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6a190a0-8198-4463-b199-dd7b7515425c-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:05.216863 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:05.216835    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a47471-2p599"
Oct 30 20:30:05.217261 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:05.216871    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55a47471-2p599" event={"ID":"a6a190a0-8198-4463-b199-dd7b7515425c","Type":"ContainerDied","Data":"d099ff06f337024505f4f087a00ce24ec6d544709ee19d0e20024e82ca15143f"}
Oct 30 20:30:05.217261 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:05.216926    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d099ff06f337024505f4f087a00ce24ec6d544709ee19d0e20024e82ca15143f"
Oct 30 20:30:06.605196 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:30:06.605165    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:30:09.982306 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:09.982261    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986" podUID="ee6329a5-2f0b-47c7-84ec-25d1c72ba396" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:30:10.302633 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:10.302545    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82" podUID="4bfb13d3-e216-41f2-996d-11541708e890" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:30:10.303013 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:10.302980    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm" podUID="f17664d0-97ab-432b-8e09-dbf95dc52b15" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:30:10.997909 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:10.997872    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6" podUID="2773ad9c-dde9-409c-937d-2a85268696f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:30:14.392148 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:14.392117    5958 scope.go:122] "RemoveContainer" containerID="71eb3c49f0103637d7f9b8a769910d5c31b8b2c9b06bff33105f3de57a11dd54"
Oct 30 20:30:15.281751 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:15.281724    5958 generic.go:356] "Generic (PLEG): container finished" podID="32b872e0-6cb5-48ef-9260-5a41099ac864" containerID="8db9ec52d87ccfc7d2e8d16bd00095da754f57b782e933a365d8a325c4cbadc9" exitCode=60
Oct 30 20:30:15.281858 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:15.281800    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz" event={"ID":"32b872e0-6cb5-48ef-9260-5a41099ac864","Type":"ContainerDied","Data":"8db9ec52d87ccfc7d2e8d16bd00095da754f57b782e933a365d8a325c4cbadc9"}
Oct 30 20:30:15.281916 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:15.281883    5958 scope.go:122] "RemoveContainer" containerID="71eb3c49f0103637d7f9b8a769910d5c31b8b2c9b06bff33105f3de57a11dd54"
Oct 30 20:30:15.282164 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:15.282144    5958 scope.go:122] "RemoveContainer" containerID="8db9ec52d87ccfc7d2e8d16bd00095da754f57b782e933a365d8a325c4cbadc9"
Oct 30 20:30:15.282365 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:30:15.282346    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889870-r8gbz_odh-ai-gateway-infra(32b872e0-6cb5-48ef-9260-5a41099ac864)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz" podUID="32b872e0-6cb5-48ef-9260-5a41099ac864"
Oct 30 20:30:15.391035 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:15.391014    5958 scope.go:122] "RemoveContainer" containerID="cd287b9ebe7870a19788c1e51312be7a5e8e8e64d25d30a9ed1caa05a33f8319"
Oct 30 20:30:16.302809 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.302767    5958 generic.go:356] "Generic (PLEG): container finished" podID="7beeda33-6b1f-40f1-a473-a5e84e2414a6" containerID="a2f42799af990cfa41eb7459404d44b0f0c5c18dd453f1c45d35073e89083def" exitCode=60
Oct 30 20:30:16.302809 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.302796    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx" event={"ID":"7beeda33-6b1f-40f1-a473-a5e84e2414a6","Type":"ContainerDied","Data":"a2f42799af990cfa41eb7459404d44b0f0c5c18dd453f1c45d35073e89083def"}
Oct 30 20:30:16.303325 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.302862    5958 scope.go:122] "RemoveContainer" containerID="cd287b9ebe7870a19788c1e51312be7a5e8e8e64d25d30a9ed1caa05a33f8319"
Oct 30 20:30:16.303325 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.303254    5958 scope.go:122] "RemoveContainer" containerID="a2f42799af990cfa41eb7459404d44b0f0c5c18dd453f1c45d35073e89083def"
Oct 30 20:30:16.303620 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:30:16.303592    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx_odh-ai-gateway-infra(7beeda33-6b1f-40f1-a473-a5e84e2414a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx" podUID="7beeda33-6b1f-40f1-a473-a5e84e2414a6"
Oct 30 20:30:16.391383 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.391360    5958 scope.go:122] "RemoveContainer" containerID="3a57a556791202b60771fb1abb64fac3351a68adcd02ba3a287e05f852fb1c49"
Oct 30 20:30:16.419050 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.419011    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz"]
Oct 30 20:30:16.613542 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.613512    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz"
Oct 30 20:30:16.674083 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.674056    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82"
Oct 30 20:30:16.699010 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.698977    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32b872e0-6cb5-48ef-9260-5a41099ac864-kube-api-access-bz879\" (UniqueName: \"kubernetes.io/projected/32b872e0-6cb5-48ef-9260-5a41099ac864-kube-api-access-bz879\") pod \"32b872e0-6cb5-48ef-9260-5a41099ac864\" (UID: \"32b872e0-6cb5-48ef-9260-5a41099ac864\") "
Oct 30 20:30:16.699159 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.699040    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32b872e0-6cb5-48ef-9260-5a41099ac864-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32b872e0-6cb5-48ef-9260-5a41099ac864-maas-api-service-ca\") pod \"32b872e0-6cb5-48ef-9260-5a41099ac864\" (UID: \"32b872e0-6cb5-48ef-9260-5a41099ac864\") "
Oct 30 20:30:16.699380 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.699359    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b872e0-6cb5-48ef-9260-5a41099ac864-maas-api-service-ca" pod "32b872e0-6cb5-48ef-9260-5a41099ac864" (UID: "32b872e0-6cb5-48ef-9260-5a41099ac864"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:16.701266 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.701245    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b872e0-6cb5-48ef-9260-5a41099ac864-kube-api-access-bz879" pod "32b872e0-6cb5-48ef-9260-5a41099ac864" (UID: "32b872e0-6cb5-48ef-9260-5a41099ac864"). InnerVolumeSpecName "kube-api-access-bz879". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:16.782908 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.782884    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm"
Oct 30 20:30:16.800335 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.800311    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bfb13d3-e216-41f2-996d-11541708e890-kube-api-access-v7xcr\" (UniqueName: \"kubernetes.io/projected/4bfb13d3-e216-41f2-996d-11541708e890-kube-api-access-v7xcr\") pod \"4bfb13d3-e216-41f2-996d-11541708e890\" (UID: \"4bfb13d3-e216-41f2-996d-11541708e890\") "
Oct 30 20:30:16.800437 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.800344    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bfb13d3-e216-41f2-996d-11541708e890-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bfb13d3-e216-41f2-996d-11541708e890-plugins-config-volume\") pod \"4bfb13d3-e216-41f2-996d-11541708e890\" (UID: \"4bfb13d3-e216-41f2-996d-11541708e890\") "
Oct 30 20:30:16.800694 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.800680    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz879\" (UniqueName: \"kubernetes.io/projected/32b872e0-6cb5-48ef-9260-5a41099ac864-kube-api-access-bz879\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:16.800765 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.800695    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32b872e0-6cb5-48ef-9260-5a41099ac864-maas-api-service-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:16.800765 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.800751    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bfb13d3-e216-41f2-996d-11541708e890-plugins-config-volume" pod "4bfb13d3-e216-41f2-996d-11541708e890" (UID: "4bfb13d3-e216-41f2-996d-11541708e890"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:16.802500 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.802480    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bfb13d3-e216-41f2-996d-11541708e890-kube-api-access-v7xcr" pod "4bfb13d3-e216-41f2-996d-11541708e890" (UID: "4bfb13d3-e216-41f2-996d-11541708e890"). InnerVolumeSpecName "kube-api-access-v7xcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:16.901632 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.901600    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f17664d0-97ab-432b-8e09-dbf95dc52b15-kube-api-access-psqld\" (UniqueName: \"kubernetes.io/projected/f17664d0-97ab-432b-8e09-dbf95dc52b15-kube-api-access-psqld\") pod \"f17664d0-97ab-432b-8e09-dbf95dc52b15\" (UID: \"f17664d0-97ab-432b-8e09-dbf95dc52b15\") "
Oct 30 20:30:16.902146 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.901716    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f17664d0-97ab-432b-8e09-dbf95dc52b15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f17664d0-97ab-432b-8e09-dbf95dc52b15-plugins-config-volume\") pod \"f17664d0-97ab-432b-8e09-dbf95dc52b15\" (UID: \"f17664d0-97ab-432b-8e09-dbf95dc52b15\") "
Oct 30 20:30:16.902146 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.902086    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f17664d0-97ab-432b-8e09-dbf95dc52b15-plugins-config-volume" pod "f17664d0-97ab-432b-8e09-dbf95dc52b15" (UID: "f17664d0-97ab-432b-8e09-dbf95dc52b15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:16.902262 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.902221    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7xcr\" (UniqueName: \"kubernetes.io/projected/4bfb13d3-e216-41f2-996d-11541708e890-kube-api-access-v7xcr\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:16.902262 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.902234    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bfb13d3-e216-41f2-996d-11541708e890-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:16.902262 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.902247    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f17664d0-97ab-432b-8e09-dbf95dc52b15-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:16.904115 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:16.904094    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f17664d0-97ab-432b-8e09-dbf95dc52b15-kube-api-access-psqld" pod "f17664d0-97ab-432b-8e09-dbf95dc52b15" (UID: "f17664d0-97ab-432b-8e09-dbf95dc52b15"). InnerVolumeSpecName "kube-api-access-psqld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:17.003698 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.003622    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psqld\" (UniqueName: \"kubernetes.io/projected/f17664d0-97ab-432b-8e09-dbf95dc52b15-kube-api-access-psqld\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:17.314656 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.314621    5958 generic.go:356] "Generic (PLEG): container finished" podID="f17664d0-97ab-432b-8e09-dbf95dc52b15" containerID="83cc183fc372d25dea8c7a9ea816929e623b0981abeeb81907b2005b201ea67d" exitCode=0
Oct 30 20:30:17.315104 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.314665    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm"
Oct 30 20:30:17.315104 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.314700    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm" event={"ID":"f17664d0-97ab-432b-8e09-dbf95dc52b15","Type":"ContainerDied","Data":"83cc183fc372d25dea8c7a9ea816929e623b0981abeeb81907b2005b201ea67d"}
Oct 30 20:30:17.315104 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.314752    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm" event={"ID":"f17664d0-97ab-432b-8e09-dbf95dc52b15","Type":"ContainerDied","Data":"19f3b104606bcffa00b3ed5381fc2516e65440a2271b1147d7fe7e9538cda6fd"}
Oct 30 20:30:17.315104 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.314774    5958 scope.go:122] "RemoveContainer" containerID="83cc183fc372d25dea8c7a9ea816929e623b0981abeeb81907b2005b201ea67d"
Oct 30 20:30:17.318161 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.318129    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz"
Oct 30 20:30:17.318276 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.318186    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz" event={"ID":"32b872e0-6cb5-48ef-9260-5a41099ac864","Type":"ContainerDied","Data":"78881d594650d85f58f2911f252b41e6467abe79e4eedf9d8883ab05b7ed0a3c"}
Oct 30 20:30:17.320521 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.320492    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw" event={"ID":"8e5a7c3a-bb1a-425e-b0db-934e55ac702f","Type":"ContainerStarted","Data":"cd1274120d2eb426ee92ad9b83df32f1e7ced0eaaad4883f88ce489070c7b0d8"}
Oct 30 20:30:17.325228 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.325206    5958 scope.go:122] "RemoveContainer" containerID="83cc183fc372d25dea8c7a9ea816929e623b0981abeeb81907b2005b201ea67d"
Oct 30 20:30:17.325538 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:30:17.325510    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83cc183fc372d25dea8c7a9ea816929e623b0981abeeb81907b2005b201ea67d\": container with ID starting with 83cc183fc372d25dea8c7a9ea816929e623b0981abeeb81907b2005b201ea67d not found: ID does not exist" containerID="83cc183fc372d25dea8c7a9ea816929e623b0981abeeb81907b2005b201ea67d"
Oct 30 20:30:17.325650 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.325549    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83cc183fc372d25dea8c7a9ea816929e623b0981abeeb81907b2005b201ea67d"} err="failed to get container status \"83cc183fc372d25dea8c7a9ea816929e623b0981abeeb81907b2005b201ea67d\": rpc error: code = NotFound desc = could not find container \"83cc183fc372d25dea8c7a9ea816929e623b0981abeeb81907b2005b201ea67d\": container with ID starting with 83cc183fc372d25dea8c7a9ea816929e623b0981abeeb81907b2005b201ea67d not found: ID does not exist"
Oct 30 20:30:17.325650 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.325573    5958 scope.go:122] "RemoveContainer" containerID="8db9ec52d87ccfc7d2e8d16bd00095da754f57b782e933a365d8a325c4cbadc9"
Oct 30 20:30:17.326558 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.326438    5958 generic.go:356] "Generic (PLEG): container finished" podID="4bfb13d3-e216-41f2-996d-11541708e890" containerID="29327240b0ad52a5e4f5030c7582e87efcc50f187c1db4488f2252c76b738775" exitCode=0
Oct 30 20:30:17.326558 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.326531    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82" event={"ID":"4bfb13d3-e216-41f2-996d-11541708e890","Type":"ContainerDied","Data":"29327240b0ad52a5e4f5030c7582e87efcc50f187c1db4488f2252c76b738775"}
Oct 30 20:30:17.326558 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.326554    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82"
Oct 30 20:30:17.326723 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.326571    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82" event={"ID":"4bfb13d3-e216-41f2-996d-11541708e890","Type":"ContainerDied","Data":"68c58e62bc7b443f30f984cbf4d406260026a309508c8c8fa00fe391631a942a"}
Oct 30 20:30:17.335147 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.335128    5958 scope.go:122] "RemoveContainer" containerID="29327240b0ad52a5e4f5030c7582e87efcc50f187c1db4488f2252c76b738775"
Oct 30 20:30:17.343726 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.343708    5958 scope.go:122] "RemoveContainer" containerID="29327240b0ad52a5e4f5030c7582e87efcc50f187c1db4488f2252c76b738775"
Oct 30 20:30:17.344043 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:30:17.344019    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29327240b0ad52a5e4f5030c7582e87efcc50f187c1db4488f2252c76b738775\": container with ID starting with 29327240b0ad52a5e4f5030c7582e87efcc50f187c1db4488f2252c76b738775 not found: ID does not exist" containerID="29327240b0ad52a5e4f5030c7582e87efcc50f187c1db4488f2252c76b738775"
Oct 30 20:30:17.344098 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.344051    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29327240b0ad52a5e4f5030c7582e87efcc50f187c1db4488f2252c76b738775"} err="failed to get container status \"29327240b0ad52a5e4f5030c7582e87efcc50f187c1db4488f2252c76b738775\": rpc error: code = NotFound desc = could not find container \"29327240b0ad52a5e4f5030c7582e87efcc50f187c1db4488f2252c76b738775\": container with ID starting with 29327240b0ad52a5e4f5030c7582e87efcc50f187c1db4488f2252c76b738775 not found: ID does not exist"
Oct 30 20:30:17.451563 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.451523    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx"]
Oct 30 20:30:17.492914 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.492866    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82"]
Oct 30 20:30:17.501039 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.501005    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-407632-96cf68f6-scr82"]
Oct 30 20:30:17.614534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.614504    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz"]
Oct 30 20:30:17.619131 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.619102    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889870-r8gbz"]
Oct 30 20:30:17.625722 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.625697    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx"
Oct 30 20:30:17.715043 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.715003    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7beeda33-6b1f-40f1-a473-a5e84e2414a6-kube-api-access-wz6fx\" (UniqueName: \"kubernetes.io/projected/7beeda33-6b1f-40f1-a473-a5e84e2414a6-kube-api-access-wz6fx\") pod \"7beeda33-6b1f-40f1-a473-a5e84e2414a6\" (UID: \"7beeda33-6b1f-40f1-a473-a5e84e2414a6\") "
Oct 30 20:30:17.715206 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.715163    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7beeda33-6b1f-40f1-a473-a5e84e2414a6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7beeda33-6b1f-40f1-a473-a5e84e2414a6-maas-api-service-ca\") pod \"7beeda33-6b1f-40f1-a473-a5e84e2414a6\" (UID: \"7beeda33-6b1f-40f1-a473-a5e84e2414a6\") "
Oct 30 20:30:17.715638 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.715602    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7beeda33-6b1f-40f1-a473-a5e84e2414a6-maas-api-service-ca" pod "7beeda33-6b1f-40f1-a473-a5e84e2414a6" (UID: "7beeda33-6b1f-40f1-a473-a5e84e2414a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:17.717222 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.717199    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7beeda33-6b1f-40f1-a473-a5e84e2414a6-kube-api-access-wz6fx" pod "7beeda33-6b1f-40f1-a473-a5e84e2414a6" (UID: "7beeda33-6b1f-40f1-a473-a5e84e2414a6"). InnerVolumeSpecName "kube-api-access-wz6fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:17.738515 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.738444    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw" podStartSLOduration=17.738426771 podStartE2EDuration="17.738426771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:17.734575956 +0000 UTC m=+2856.466791356" watchObservedRunningTime="2026-10-30 20:30:17.738426771 +0000 UTC m=+2856.470642160"
Oct 30 20:30:17.816902 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.816864    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7beeda33-6b1f-40f1-a473-a5e84e2414a6-maas-api-service-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:17.816902 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.816898    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz6fx\" (UniqueName: \"kubernetes.io/projected/7beeda33-6b1f-40f1-a473-a5e84e2414a6-kube-api-access-wz6fx\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:17.847062 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.847031    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm"]
Oct 30 20:30:17.851507 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:17.851434    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-407632-5895bc8cc5-fj5cm"]
Oct 30 20:30:18.217491 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:18.217399    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-14c846-jwww4"]
Oct 30 20:30:18.220817 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:18.220788    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-14c846-jwww4"]
Oct 30 20:30:18.335040 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:18.335010    5958 generic.go:356] "Generic (PLEG): container finished" podID="8e5a7c3a-bb1a-425e-b0db-934e55ac702f" containerID="cd1274120d2eb426ee92ad9b83df32f1e7ced0eaaad4883f88ce489070c7b0d8" exitCode=60
Oct 30 20:30:18.335523 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:18.335079    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw" event={"ID":"8e5a7c3a-bb1a-425e-b0db-934e55ac702f","Type":"ContainerDied","Data":"cd1274120d2eb426ee92ad9b83df32f1e7ced0eaaad4883f88ce489070c7b0d8"}
Oct 30 20:30:18.335523 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:18.335148    5958 scope.go:122] "RemoveContainer" containerID="3a57a556791202b60771fb1abb64fac3351a68adcd02ba3a287e05f852fb1c49"
Oct 30 20:30:18.335666 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:18.335553    5958 scope.go:122] "RemoveContainer" containerID="cd1274120d2eb426ee92ad9b83df32f1e7ced0eaaad4883f88ce489070c7b0d8"
Oct 30 20:30:18.335855 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:30:18.335834    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw_odh-ai-gateway-infra(8e5a7c3a-bb1a-425e-b0db-934e55ac702f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw" podUID="8e5a7c3a-bb1a-425e-b0db-934e55ac702f"
Oct 30 20:30:18.338989 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:18.338966    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx" event={"ID":"7beeda33-6b1f-40f1-a473-a5e84e2414a6","Type":"ContainerDied","Data":"9397c2e23fb7480f8443ccfbc2acbf83dae1e417a7af8b4503209596fb68723a"}
Oct 30 20:30:18.339099 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:18.339001    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx"
Oct 30 20:30:18.351753 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:18.351618    5958 scope.go:122] "RemoveContainer" containerID="a2f42799af990cfa41eb7459404d44b0f0c5c18dd453f1c45d35073e89083def"
Oct 30 20:30:18.596294 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:18.596233    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx"]
Oct 30 20:30:18.606753 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:18.606714    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3ccc9f-29889870-skfqx"]
Oct 30 20:30:18.745916 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:18.745883    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw"]
Oct 30 20:30:19.397583 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:19.397552    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32b872e0-6cb5-48ef-9260-5a41099ac864" path="/var/lib/kubelet/pods/32b872e0-6cb5-48ef-9260-5a41099ac864/volumes"
Oct 30 20:30:19.398038 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:19.397944    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bfb13d3-e216-41f2-996d-11541708e890" path="/var/lib/kubelet/pods/4bfb13d3-e216-41f2-996d-11541708e890/volumes"
Oct 30 20:30:19.398845 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:19.398398    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7beeda33-6b1f-40f1-a473-a5e84e2414a6" path="/var/lib/kubelet/pods/7beeda33-6b1f-40f1-a473-a5e84e2414a6/volumes"
Oct 30 20:30:19.399586 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:19.399482    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf455d6d-8843-4c51-bf9d-07135b9577e8" path="/var/lib/kubelet/pods/bf455d6d-8843-4c51-bf9d-07135b9577e8/volumes"
Oct 30 20:30:19.399991 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:19.399966    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f17664d0-97ab-432b-8e09-dbf95dc52b15" path="/var/lib/kubelet/pods/f17664d0-97ab-432b-8e09-dbf95dc52b15/volumes"
Oct 30 20:30:19.481797 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:19.481774    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw"
Oct 30 20:30:19.636851 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:19.636764    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e5a7c3a-bb1a-425e-b0db-934e55ac702f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5a7c3a-bb1a-425e-b0db-934e55ac702f-maas-api-service-ca\") pod \"8e5a7c3a-bb1a-425e-b0db-934e55ac702f\" (UID: \"8e5a7c3a-bb1a-425e-b0db-934e55ac702f\") "
Oct 30 20:30:19.637017 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:19.636856    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e5a7c3a-bb1a-425e-b0db-934e55ac702f-kube-api-access-lsqpl\" (UniqueName: \"kubernetes.io/projected/8e5a7c3a-bb1a-425e-b0db-934e55ac702f-kube-api-access-lsqpl\") pod \"8e5a7c3a-bb1a-425e-b0db-934e55ac702f\" (UID: \"8e5a7c3a-bb1a-425e-b0db-934e55ac702f\") "
Oct 30 20:30:19.637184 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:19.637158    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5a7c3a-bb1a-425e-b0db-934e55ac702f-maas-api-service-ca" pod "8e5a7c3a-bb1a-425e-b0db-934e55ac702f" (UID: "8e5a7c3a-bb1a-425e-b0db-934e55ac702f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:19.639168 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:19.639139    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5a7c3a-bb1a-425e-b0db-934e55ac702f-kube-api-access-lsqpl" pod "8e5a7c3a-bb1a-425e-b0db-934e55ac702f" (UID: "8e5a7c3a-bb1a-425e-b0db-934e55ac702f"). InnerVolumeSpecName "kube-api-access-lsqpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:19.738640 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:19.738608    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e5a7c3a-bb1a-425e-b0db-934e55ac702f-maas-api-service-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:19.738640 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:19.738637    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsqpl\" (UniqueName: \"kubernetes.io/projected/8e5a7c3a-bb1a-425e-b0db-934e55ac702f-kube-api-access-lsqpl\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:19.982602 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:19.982518    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986" podUID="ee6329a5-2f0b-47c7-84ec-25d1c72ba396" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:30:20.354243 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:20.354215    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw"
Oct 30 20:30:20.354415 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:20.354253    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw" event={"ID":"8e5a7c3a-bb1a-425e-b0db-934e55ac702f","Type":"ContainerDied","Data":"7a9f5dda9b076416e16ccdb4b80d577dca2fad019b186489eeefdd811e40fde5"}
Oct 30 20:30:20.354415 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:20.354318    5958 scope.go:122] "RemoveContainer" containerID="cd1274120d2eb426ee92ad9b83df32f1e7ced0eaaad4883f88ce489070c7b0d8"
Oct 30 20:30:20.520061 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:20.520024    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw"]
Oct 30 20:30:20.569951 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:20.569923    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72db69-29889870-jbqkw"]
Oct 30 20:30:20.997259 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:20.997228    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6" podUID="2773ad9c-dde9-409c-937d-2a85268696f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:30:21.397816 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:21.397784    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e5a7c3a-bb1a-425e-b0db-934e55ac702f" path="/var/lib/kubelet/pods/8e5a7c3a-bb1a-425e-b0db-934e55ac702f/volumes"
Oct 30 20:30:24.379096 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:24.379067    5958 generic.go:356] "Generic (PLEG): container finished" podID="2773ad9c-dde9-409c-937d-2a85268696f1" containerID="824047327c083de6da5d33c09ff11b0f9fa1aba65fb5b65a16b059097e4c4ba8" exitCode=0
Oct 30 20:30:24.379396 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:24.379138    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6" event={"ID":"2773ad9c-dde9-409c-937d-2a85268696f1","Type":"ContainerDied","Data":"824047327c083de6da5d33c09ff11b0f9fa1aba65fb5b65a16b059097e4c4ba8"}
Oct 30 20:30:24.542912 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:24.542889    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6"
Oct 30 20:30:24.627531 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:24.627507    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986"
Oct 30 20:30:24.648232 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:24.648204    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-407632-d494w"]
Oct 30 20:30:24.649880 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:24.649856    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-407632-d494w"]
Oct 30 20:30:24.689380 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:24.689357    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2773ad9c-dde9-409c-937d-2a85268696f1-kube-api-access-tvh5l\" (UniqueName: \"kubernetes.io/projected/2773ad9c-dde9-409c-937d-2a85268696f1-kube-api-access-tvh5l\") pod \"2773ad9c-dde9-409c-937d-2a85268696f1\" (UID: \"2773ad9c-dde9-409c-937d-2a85268696f1\") "
Oct 30 20:30:24.689499 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:24.689424    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2773ad9c-dde9-409c-937d-2a85268696f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2773ad9c-dde9-409c-937d-2a85268696f1-plugins-config-volume\") pod \"2773ad9c-dde9-409c-937d-2a85268696f1\" (UID: \"2773ad9c-dde9-409c-937d-2a85268696f1\") "
Oct 30 20:30:24.689774 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:24.689753    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2773ad9c-dde9-409c-937d-2a85268696f1-plugins-config-volume" pod "2773ad9c-dde9-409c-937d-2a85268696f1" (UID: "2773ad9c-dde9-409c-937d-2a85268696f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:24.691478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:24.691458    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2773ad9c-dde9-409c-937d-2a85268696f1-kube-api-access-tvh5l" pod "2773ad9c-dde9-409c-937d-2a85268696f1" (UID: "2773ad9c-dde9-409c-937d-2a85268696f1"). InnerVolumeSpecName "kube-api-access-tvh5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:24.790731 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:24.790659    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee6329a5-2f0b-47c7-84ec-25d1c72ba396-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee6329a5-2f0b-47c7-84ec-25d1c72ba396-plugins-config-volume\") pod \"ee6329a5-2f0b-47c7-84ec-25d1c72ba396\" (UID: \"ee6329a5-2f0b-47c7-84ec-25d1c72ba396\") "
Oct 30 20:30:24.790905 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:24.790734    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee6329a5-2f0b-47c7-84ec-25d1c72ba396-kube-api-access-dvq8z\" (UniqueName: \"kubernetes.io/projected/ee6329a5-2f0b-47c7-84ec-25d1c72ba396-kube-api-access-dvq8z\") pod \"ee6329a5-2f0b-47c7-84ec-25d1c72ba396\" (UID: \"ee6329a5-2f0b-47c7-84ec-25d1c72ba396\") "
Oct 30 20:30:24.791046 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:24.791021    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee6329a5-2f0b-47c7-84ec-25d1c72ba396-plugins-config-volume" pod "ee6329a5-2f0b-47c7-84ec-25d1c72ba396" (UID: "ee6329a5-2f0b-47c7-84ec-25d1c72ba396"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:24.791113 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:24.791098    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2773ad9c-dde9-409c-937d-2a85268696f1-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:24.791164 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:24.791121    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvh5l\" (UniqueName: \"kubernetes.io/projected/2773ad9c-dde9-409c-937d-2a85268696f1-kube-api-access-tvh5l\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:24.792902 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:24.792881    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee6329a5-2f0b-47c7-84ec-25d1c72ba396-kube-api-access-dvq8z" pod "ee6329a5-2f0b-47c7-84ec-25d1c72ba396" (UID: "ee6329a5-2f0b-47c7-84ec-25d1c72ba396"). InnerVolumeSpecName "kube-api-access-dvq8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:24.892722 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:24.892689    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee6329a5-2f0b-47c7-84ec-25d1c72ba396-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:24.892722 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:24.892715    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvq8z\" (UniqueName: \"kubernetes.io/projected/ee6329a5-2f0b-47c7-84ec-25d1c72ba396-kube-api-access-dvq8z\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:25.386293 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:25.386258    5958 generic.go:356] "Generic (PLEG): container finished" podID="ee6329a5-2f0b-47c7-84ec-25d1c72ba396" containerID="5e9aab54d9a95dd3a734df96406946af48a01179ad8629fa21fab3ebc656e95d" exitCode=0
Oct 30 20:30:25.386821 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:25.386300    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986"
Oct 30 20:30:25.386821 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:25.386379    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986" event={"ID":"ee6329a5-2f0b-47c7-84ec-25d1c72ba396","Type":"ContainerDied","Data":"5e9aab54d9a95dd3a734df96406946af48a01179ad8629fa21fab3ebc656e95d"}
Oct 30 20:30:25.386821 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:25.386441    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986" event={"ID":"ee6329a5-2f0b-47c7-84ec-25d1c72ba396","Type":"ContainerDied","Data":"dc1fdc1b5453870429a4ebb66ce8ac1b60def463b325742c300b0b54f6aba95f"}
Oct 30 20:30:25.387195 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:25.386489    5958 scope.go:122] "RemoveContainer" containerID="5e9aab54d9a95dd3a734df96406946af48a01179ad8629fa21fab3ebc656e95d"
Oct 30 20:30:25.388136 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:25.388109    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6" event={"ID":"2773ad9c-dde9-409c-937d-2a85268696f1","Type":"ContainerDied","Data":"2987548a5957e68603f6341798fb5d8894efaa21d21891681b6f12e8420f25f0"}
Oct 30 20:30:25.388228 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:25.388116    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6"
Oct 30 20:30:25.403109 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:25.403085    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db45d07e-8e54-4b01-abb2-31e7ad9fd864" path="/var/lib/kubelet/pods/db45d07e-8e54-4b01-abb2-31e7ad9fd864/volumes"
Oct 30 20:30:25.403427 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:25.403405    5958 scope.go:122] "RemoveContainer" containerID="5e9aab54d9a95dd3a734df96406946af48a01179ad8629fa21fab3ebc656e95d"
Oct 30 20:30:25.403741 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:30:25.403721    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e9aab54d9a95dd3a734df96406946af48a01179ad8629fa21fab3ebc656e95d\": container with ID starting with 5e9aab54d9a95dd3a734df96406946af48a01179ad8629fa21fab3ebc656e95d not found: ID does not exist" containerID="5e9aab54d9a95dd3a734df96406946af48a01179ad8629fa21fab3ebc656e95d"
Oct 30 20:30:25.403820 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:25.403749    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e9aab54d9a95dd3a734df96406946af48a01179ad8629fa21fab3ebc656e95d"} err="failed to get container status \"5e9aab54d9a95dd3a734df96406946af48a01179ad8629fa21fab3ebc656e95d\": rpc error: code = NotFound desc = could not find container \"5e9aab54d9a95dd3a734df96406946af48a01179ad8629fa21fab3ebc656e95d\": container with ID starting with 5e9aab54d9a95dd3a734df96406946af48a01179ad8629fa21fab3ebc656e95d not found: ID does not exist"
Oct 30 20:30:25.403820 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:25.403765    5958 scope.go:122] "RemoveContainer" containerID="824047327c083de6da5d33c09ff11b0f9fa1aba65fb5b65a16b059097e4c4ba8"
Oct 30 20:30:25.551129 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:25.551046    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6"]
Oct 30 20:30:25.556347 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:25.556323    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-14c846-896d77c4d-vr7b6"]
Oct 30 20:30:25.684778 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:25.684751    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986"]
Oct 30 20:30:25.696496 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:25.696467    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-14c846-6dc4dbcf7f-9p986"]
Oct 30 20:30:26.543822 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.543789    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72db69-h7knh"]
Oct 30 20:30:26.544275 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544260    5958 cpu_manager.go:414] "containerMap: removing container" podUID="2773ad9c-dde9-409c-937d-2a85268696f1" containerName="payload-processing"
Oct 30 20:30:26.544333 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544276    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2773ad9c-dde9-409c-937d-2a85268696f1" containerName="payload-processing"
Oct 30 20:30:26.544333 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544290    5958 cpu_manager.go:414] "containerMap: removing container" podUID="32b872e0-6cb5-48ef-9260-5a41099ac864" containerName="cleanup"
Oct 30 20:30:26.544333 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544295    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b872e0-6cb5-48ef-9260-5a41099ac864" containerName="cleanup"
Oct 30 20:30:26.544333 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544319    5958 cpu_manager.go:414] "containerMap: removing container" podUID="f17664d0-97ab-432b-8e09-dbf95dc52b15" containerName="payload-pre-processing"
Oct 30 20:30:26.544333 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544324    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f17664d0-97ab-432b-8e09-dbf95dc52b15" containerName="payload-pre-processing"
Oct 30 20:30:26.544333 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544331    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ee6329a5-2f0b-47c7-84ec-25d1c72ba396" containerName="payload-pre-processing"
Oct 30 20:30:26.544553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544336    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee6329a5-2f0b-47c7-84ec-25d1c72ba396" containerName="payload-pre-processing"
Oct 30 20:30:26.544553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544344    5958 cpu_manager.go:414] "containerMap: removing container" podUID="4bfb13d3-e216-41f2-996d-11541708e890" containerName="payload-processing"
Oct 30 20:30:26.544553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544349    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bfb13d3-e216-41f2-996d-11541708e890" containerName="payload-processing"
Oct 30 20:30:26.544553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544356    5958 cpu_manager.go:414] "containerMap: removing container" podUID="a6a190a0-8198-4463-b199-dd7b7515425c" containerName="revoke-keys"
Oct 30 20:30:26.544553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544360    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6a190a0-8198-4463-b199-dd7b7515425c" containerName="revoke-keys"
Oct 30 20:30:26.544553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544367    5958 cpu_manager.go:414] "containerMap: removing container" podUID="8e5a7c3a-bb1a-425e-b0db-934e55ac702f" containerName="cleanup"
Oct 30 20:30:26.544553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544371    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e5a7c3a-bb1a-425e-b0db-934e55ac702f" containerName="cleanup"
Oct 30 20:30:26.544553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544384    5958 cpu_manager.go:414] "containerMap: removing container" podUID="7beeda33-6b1f-40f1-a473-a5e84e2414a6" containerName="cleanup"
Oct 30 20:30:26.544553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544391    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7beeda33-6b1f-40f1-a473-a5e84e2414a6" containerName="cleanup"
Oct 30 20:30:26.544553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544399    5958 cpu_manager.go:414] "containerMap: removing container" podUID="7beeda33-6b1f-40f1-a473-a5e84e2414a6" containerName="cleanup"
Oct 30 20:30:26.544553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544404    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7beeda33-6b1f-40f1-a473-a5e84e2414a6" containerName="cleanup"
Oct 30 20:30:26.544553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544410    5958 cpu_manager.go:414] "containerMap: removing container" podUID="8e5a7c3a-bb1a-425e-b0db-934e55ac702f" containerName="cleanup"
Oct 30 20:30:26.544553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544415    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e5a7c3a-bb1a-425e-b0db-934e55ac702f" containerName="cleanup"
Oct 30 20:30:26.544553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544424    5958 cpu_manager.go:414] "containerMap: removing container" podUID="7beeda33-6b1f-40f1-a473-a5e84e2414a6" containerName="cleanup"
Oct 30 20:30:26.544553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544428    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7beeda33-6b1f-40f1-a473-a5e84e2414a6" containerName="cleanup"
Oct 30 20:30:26.544553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544434    5958 cpu_manager.go:414] "containerMap: removing container" podUID="8e5a7c3a-bb1a-425e-b0db-934e55ac702f" containerName="cleanup"
Oct 30 20:30:26.544553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544438    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e5a7c3a-bb1a-425e-b0db-934e55ac702f" containerName="cleanup"
Oct 30 20:30:26.544553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544469    5958 cpu_manager.go:414] "containerMap: removing container" podUID="32b872e0-6cb5-48ef-9260-5a41099ac864" containerName="cleanup"
Oct 30 20:30:26.544553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544476    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b872e0-6cb5-48ef-9260-5a41099ac864" containerName="cleanup"
Oct 30 20:30:26.544553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544546    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="f17664d0-97ab-432b-8e09-dbf95dc52b15" containerName="payload-pre-processing"
Oct 30 20:30:26.544553 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544557    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee6329a5-2f0b-47c7-84ec-25d1c72ba396" containerName="payload-pre-processing"
Oct 30 20:30:26.545191 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544563    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bfb13d3-e216-41f2-996d-11541708e890" containerName="payload-processing"
Oct 30 20:30:26.545191 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544570    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6a190a0-8198-4463-b199-dd7b7515425c" containerName="revoke-keys"
Oct 30 20:30:26.545191 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544574    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e5a7c3a-bb1a-425e-b0db-934e55ac702f" containerName="cleanup"
Oct 30 20:30:26.545191 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544581    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="7beeda33-6b1f-40f1-a473-a5e84e2414a6" containerName="cleanup"
Oct 30 20:30:26.545191 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544587    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="7beeda33-6b1f-40f1-a473-a5e84e2414a6" containerName="cleanup"
Oct 30 20:30:26.545191 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544592    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e5a7c3a-bb1a-425e-b0db-934e55ac702f" containerName="cleanup"
Oct 30 20:30:26.545191 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544598    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="7beeda33-6b1f-40f1-a473-a5e84e2414a6" containerName="cleanup"
Oct 30 20:30:26.545191 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544604    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b872e0-6cb5-48ef-9260-5a41099ac864" containerName="cleanup"
Oct 30 20:30:26.545191 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544610    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b872e0-6cb5-48ef-9260-5a41099ac864" containerName="cleanup"
Oct 30 20:30:26.545191 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544618    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="2773ad9c-dde9-409c-937d-2a85268696f1" containerName="payload-processing"
Oct 30 20:30:26.545191 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.544627    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b872e0-6cb5-48ef-9260-5a41099ac864" containerName="cleanup"
Oct 30 20:30:26.549267 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.549249    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72db69-h7knh"
Oct 30 20:30:26.604074 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.604048    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:30:26.604178 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.604115    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:30:26.612518 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.612500    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:30:26.612582 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.612537    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:30:26.654081 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.654055    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72db69-h7knh"]
Oct 30 20:30:26.708551 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.708523    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdhjj\" (UniqueName: \"kubernetes.io/projected/3671222c-4562-4e2d-ab26-39aeff8b0c6f-kube-api-access-pdhjj\") pod \"maas-api-revoke-keys-e2e-inf-b-72db69-h7knh\" (UID: \"3671222c-4562-4e2d-ab26-39aeff8b0c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72db69-h7knh"
Oct 30 20:30:26.708659 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.708590    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3671222c-4562-4e2d-ab26-39aeff8b0c6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-72db69-h7knh\" (UID: \"3671222c-4562-4e2d-ab26-39aeff8b0c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72db69-h7knh"
Oct 30 20:30:26.708750 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.708732    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3671222c-4562-4e2d-ab26-39aeff8b0c6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-72db69-h7knh\" (UID: \"3671222c-4562-4e2d-ab26-39aeff8b0c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72db69-h7knh"
Oct 30 20:30:26.810128 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.810065    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3671222c-4562-4e2d-ab26-39aeff8b0c6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-72db69-h7knh\" (UID: \"3671222c-4562-4e2d-ab26-39aeff8b0c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72db69-h7knh"
Oct 30 20:30:26.810244 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.810147    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3671222c-4562-4e2d-ab26-39aeff8b0c6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-72db69-h7knh\" (UID: \"3671222c-4562-4e2d-ab26-39aeff8b0c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72db69-h7knh"
Oct 30 20:30:26.810244 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.810192    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdhjj\" (UniqueName: \"kubernetes.io/projected/3671222c-4562-4e2d-ab26-39aeff8b0c6f-kube-api-access-pdhjj\") pod \"maas-api-revoke-keys-e2e-inf-b-72db69-h7knh\" (UID: \"3671222c-4562-4e2d-ab26-39aeff8b0c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72db69-h7knh"
Oct 30 20:30:26.810752 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.810731    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3671222c-4562-4e2d-ab26-39aeff8b0c6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-72db69-h7knh\" (UID: \"3671222c-4562-4e2d-ab26-39aeff8b0c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72db69-h7knh"
Oct 30 20:30:26.812531 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.812512    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3671222c-4562-4e2d-ab26-39aeff8b0c6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-72db69-h7knh\" (UID: \"3671222c-4562-4e2d-ab26-39aeff8b0c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72db69-h7knh"
Oct 30 20:30:26.870821 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:26.870794    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdhjj\" (UniqueName: \"kubernetes.io/projected/3671222c-4562-4e2d-ab26-39aeff8b0c6f-kube-api-access-pdhjj\") pod \"maas-api-revoke-keys-e2e-inf-b-72db69-h7knh\" (UID: \"3671222c-4562-4e2d-ab26-39aeff8b0c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72db69-h7knh"
Oct 30 20:30:27.161610 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:27.161578    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72db69-h7knh"
Oct 30 20:30:27.397257 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:27.397228    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2773ad9c-dde9-409c-937d-2a85268696f1" path="/var/lib/kubelet/pods/2773ad9c-dde9-409c-937d-2a85268696f1/volumes"
Oct 30 20:30:27.397655 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:27.397640    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee6329a5-2f0b-47c7-84ec-25d1c72ba396" path="/var/lib/kubelet/pods/ee6329a5-2f0b-47c7-84ec-25d1c72ba396/volumes"
Oct 30 20:30:27.535432 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:27.535410    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72db69-h7knh"]
Oct 30 20:30:27.537644 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:30:27.537619    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3671222c_4562_4e2d_ab26_39aeff8b0c6f.slice/crio-9113e99f488aece7d698b6e408b29e05d6fffe43bc4e8cf794c561e60135d322 WatchSource:0}: Error finding container 9113e99f488aece7d698b6e408b29e05d6fffe43bc4e8cf794c561e60135d322: Status 404 returned error can't find the container with id 9113e99f488aece7d698b6e408b29e05d6fffe43bc4e8cf794c561e60135d322
Oct 30 20:30:28.407070 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:28.407033    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72db69-h7knh" event={"ID":"3671222c-4562-4e2d-ab26-39aeff8b0c6f","Type":"ContainerStarted","Data":"551cc56a5fb8fd5675c10da4b69324919ceaeb531d4dda12f9c0efdb1ee20962"}
Oct 30 20:30:28.409537 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:28.407095    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72db69-h7knh" event={"ID":"3671222c-4562-4e2d-ab26-39aeff8b0c6f","Type":"ContainerStarted","Data":"9113e99f488aece7d698b6e408b29e05d6fffe43bc4e8cf794c561e60135d322"}
Oct 30 20:30:28.526542 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:28.526489    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72db69-h7knh" podStartSLOduration=2.526476196 podStartE2EDuration="2.526476196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:28.52394058 +0000 UTC m=+2867.256155966" watchObservedRunningTime="2026-10-30 20:30:28.526476196 +0000 UTC m=+2867.258691586"
Oct 30 20:30:29.413031 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:29.413000    5958 generic.go:356] "Generic (PLEG): container finished" podID="3671222c-4562-4e2d-ab26-39aeff8b0c6f" containerID="551cc56a5fb8fd5675c10da4b69324919ceaeb531d4dda12f9c0efdb1ee20962" exitCode=0
Oct 30 20:30:29.413407 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:29.413080    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72db69-h7knh" event={"ID":"3671222c-4562-4e2d-ab26-39aeff8b0c6f","Type":"ContainerDied","Data":"551cc56a5fb8fd5675c10da4b69324919ceaeb531d4dda12f9c0efdb1ee20962"}
Oct 30 20:30:30.551989 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:30.551962    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72db69-h7knh"
Oct 30 20:30:30.644769 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:30.644739    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3671222c-4562-4e2d-ab26-39aeff8b0c6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3671222c-4562-4e2d-ab26-39aeff8b0c6f-maas-api-xks-ca\") pod \"3671222c-4562-4e2d-ab26-39aeff8b0c6f\" (UID: \"3671222c-4562-4e2d-ab26-39aeff8b0c6f\") "
Oct 30 20:30:30.644890 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:30.644794    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3671222c-4562-4e2d-ab26-39aeff8b0c6f-kube-api-access-pdhjj\" (UniqueName: \"kubernetes.io/projected/3671222c-4562-4e2d-ab26-39aeff8b0c6f-kube-api-access-pdhjj\") pod \"3671222c-4562-4e2d-ab26-39aeff8b0c6f\" (UID: \"3671222c-4562-4e2d-ab26-39aeff8b0c6f\") "
Oct 30 20:30:30.644978 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:30.644961    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3671222c-4562-4e2d-ab26-39aeff8b0c6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3671222c-4562-4e2d-ab26-39aeff8b0c6f-maas-api-openshift-ca\") pod \"3671222c-4562-4e2d-ab26-39aeff8b0c6f\" (UID: \"3671222c-4562-4e2d-ab26-39aeff8b0c6f\") "
Oct 30 20:30:30.645297 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:30.645270    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3671222c-4562-4e2d-ab26-39aeff8b0c6f-maas-api-openshift-ca" pod "3671222c-4562-4e2d-ab26-39aeff8b0c6f" (UID: "3671222c-4562-4e2d-ab26-39aeff8b0c6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:30.647000 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:30.646970    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3671222c-4562-4e2d-ab26-39aeff8b0c6f-kube-api-access-pdhjj" pod "3671222c-4562-4e2d-ab26-39aeff8b0c6f" (UID: "3671222c-4562-4e2d-ab26-39aeff8b0c6f"). InnerVolumeSpecName "kube-api-access-pdhjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:30.647087 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:30.647033    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3671222c-4562-4e2d-ab26-39aeff8b0c6f-maas-api-xks-ca" pod "3671222c-4562-4e2d-ab26-39aeff8b0c6f" (UID: "3671222c-4562-4e2d-ab26-39aeff8b0c6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:30:30.745926 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:30.745849    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3671222c-4562-4e2d-ab26-39aeff8b0c6f-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:30.745926 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:30.745879    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3671222c-4562-4e2d-ab26-39aeff8b0c6f-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:30.745926 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:30.745889    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdhjj\" (UniqueName: \"kubernetes.io/projected/3671222c-4562-4e2d-ab26-39aeff8b0c6f-kube-api-access-pdhjj\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:31.427340 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:31.427309    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72db69-h7knh"
Oct 30 20:30:31.427508 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:31.427358    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72db69-h7knh" event={"ID":"3671222c-4562-4e2d-ab26-39aeff8b0c6f","Type":"ContainerDied","Data":"9113e99f488aece7d698b6e408b29e05d6fffe43bc4e8cf794c561e60135d322"}
Oct 30 20:30:31.427508 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:31.427413    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9113e99f488aece7d698b6e408b29e05d6fffe43bc4e8cf794c561e60135d322"
Oct 30 20:30:33.027917 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:33.027881    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"]
Oct 30 20:30:33.028355 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:33.028076    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v" podUID="fde4d62b-bd67-4915-a145-e5511e651c25" containerName="istio-proxy" containerID="cri-o://096bdd50d9cdf384afa596e1c28c74943640d61fdae6fc60613abbf3eb8082a7" gracePeriod=30
Oct 30 20:30:36.605114 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:30:36.605080    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:30:38.229795 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.229760    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf"]
Oct 30 20:30:38.282364 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.282338    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 20:30:38.316852 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.316831    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-workload-certs\") pod \"fde4d62b-bd67-4915-a145-e5511e651c25\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") "
Oct 30 20:30:38.316969 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.316874    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fde4d62b-bd67-4915-a145-e5511e651c25-kube-api-access-jftrw\" (UniqueName: \"kubernetes.io/projected/fde4d62b-bd67-4915-a145-e5511e651c25-kube-api-access-jftrw\") pod \"fde4d62b-bd67-4915-a145-e5511e651c25\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") "
Oct 30 20:30:38.316969 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.316891    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fde4d62b-bd67-4915-a145-e5511e651c25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fde4d62b-bd67-4915-a145-e5511e651c25-istiod-ca-cert\") pod \"fde4d62b-bd67-4915-a145-e5511e651c25\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") "
Oct 30 20:30:38.316969 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.316920    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-istio-envoy\") pod \"fde4d62b-bd67-4915-a145-e5511e651c25\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") "
Oct 30 20:30:38.317153 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.316988    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-istio-data\") pod \"fde4d62b-bd67-4915-a145-e5511e651c25\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") "
Oct 30 20:30:38.317153 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.317030    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-workload-socket\") pod \"fde4d62b-bd67-4915-a145-e5511e651c25\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") "
Oct 30 20:30:38.317153 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.317080    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fde4d62b-bd67-4915-a145-e5511e651c25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fde4d62b-bd67-4915-a145-e5511e651c25-istio-podinfo\") pod \"fde4d62b-bd67-4915-a145-e5511e651c25\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") "
Oct 30 20:30:38.317153 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.317105    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-workload-certs" pod "fde4d62b-bd67-4915-a145-e5511e651c25" (UID: "fde4d62b-bd67-4915-a145-e5511e651c25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:38.317373 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.317181    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fde4d62b-bd67-4915-a145-e5511e651c25-istio-token\" (UniqueName: \"kubernetes.io/projected/fde4d62b-bd67-4915-a145-e5511e651c25-istio-token\") pod \"fde4d62b-bd67-4915-a145-e5511e651c25\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") "
Oct 30 20:30:38.317373 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.317216    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-credential-socket\") pod \"fde4d62b-bd67-4915-a145-e5511e651c25\" (UID: \"fde4d62b-bd67-4915-a145-e5511e651c25\") "
Oct 30 20:30:38.317373 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.317301    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-workload-socket" pod "fde4d62b-bd67-4915-a145-e5511e651c25" (UID: "fde4d62b-bd67-4915-a145-e5511e651c25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:38.317373 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.317311    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-istio-data" pod "fde4d62b-bd67-4915-a145-e5511e651c25" (UID: "fde4d62b-bd67-4915-a145-e5511e651c25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:38.317780 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.317575    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-credential-socket" pod "fde4d62b-bd67-4915-a145-e5511e651c25" (UID: "fde4d62b-bd67-4915-a145-e5511e651c25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:38.317780 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.317611    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fde4d62b-bd67-4915-a145-e5511e651c25-istiod-ca-cert" pod "fde4d62b-bd67-4915-a145-e5511e651c25" (UID: "fde4d62b-bd67-4915-a145-e5511e651c25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:38.317957 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.317830    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:38.317957 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.317854    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:38.317957 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.317906    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:38.317957 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.317929    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:38.319406 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.319381    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde4d62b-bd67-4915-a145-e5511e651c25-kube-api-access-jftrw" pod "fde4d62b-bd67-4915-a145-e5511e651c25" (UID: "fde4d62b-bd67-4915-a145-e5511e651c25"). InnerVolumeSpecName "kube-api-access-jftrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:38.319538 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.319433    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-istio-envoy" pod "fde4d62b-bd67-4915-a145-e5511e651c25" (UID: "fde4d62b-bd67-4915-a145-e5511e651c25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:38.319863 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.319847    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde4d62b-bd67-4915-a145-e5511e651c25-istio-token" pod "fde4d62b-bd67-4915-a145-e5511e651c25" (UID: "fde4d62b-bd67-4915-a145-e5511e651c25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:38.319939 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.319919    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fde4d62b-bd67-4915-a145-e5511e651c25-istio-podinfo" pod "fde4d62b-bd67-4915-a145-e5511e651c25" (UID: "fde4d62b-bd67-4915-a145-e5511e651c25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:30:38.419510 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.419486    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fde4d62b-bd67-4915-a145-e5511e651c25-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:38.419648 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.419624    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jftrw\" (UniqueName: \"kubernetes.io/projected/fde4d62b-bd67-4915-a145-e5511e651c25-kube-api-access-jftrw\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:38.419695 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.419657    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fde4d62b-bd67-4915-a145-e5511e651c25-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:38.419695 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.419669    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fde4d62b-bd67-4915-a145-e5511e651c25-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:38.419695 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.419677    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fde4d62b-bd67-4915-a145-e5511e651c25-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:38.465870 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.465839    5958 generic.go:356] "Generic (PLEG): container finished" podID="fde4d62b-bd67-4915-a145-e5511e651c25" containerID="096bdd50d9cdf384afa596e1c28c74943640d61fdae6fc60613abbf3eb8082a7" exitCode=0
Oct 30 20:30:38.466001 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.465900    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"
Oct 30 20:30:38.466001 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.465924    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v" event={"ID":"fde4d62b-bd67-4915-a145-e5511e651c25","Type":"ContainerDied","Data":"096bdd50d9cdf384afa596e1c28c74943640d61fdae6fc60613abbf3eb8082a7"}
Oct 30 20:30:38.466001 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.465989    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v" event={"ID":"fde4d62b-bd67-4915-a145-e5511e651c25","Type":"ContainerDied","Data":"08a99d775bb3cf2a4815c760d35c5bddbea0b085db900367d9b5b3884fdd2455"}
Oct 30 20:30:38.466169 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.466019    5958 scope.go:122] "RemoveContainer" containerID="096bdd50d9cdf384afa596e1c28c74943640d61fdae6fc60613abbf3eb8082a7"
Oct 30 20:30:38.478078 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.478057    5958 scope.go:122] "RemoveContainer" containerID="5804a85bb63e293e5e14f36db543f910ccec895a02f22cc2da98da5a2fcc5d16"
Oct 30 20:30:38.489802 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.489777    5958 scope.go:122] "RemoveContainer" containerID="096bdd50d9cdf384afa596e1c28c74943640d61fdae6fc60613abbf3eb8082a7"
Oct 30 20:30:38.490119 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:30:38.490099    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"096bdd50d9cdf384afa596e1c28c74943640d61fdae6fc60613abbf3eb8082a7\": container with ID starting with 096bdd50d9cdf384afa596e1c28c74943640d61fdae6fc60613abbf3eb8082a7 not found: ID does not exist" containerID="096bdd50d9cdf384afa596e1c28c74943640d61fdae6fc60613abbf3eb8082a7"
Oct 30 20:30:38.490197 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.490129    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"096bdd50d9cdf384afa596e1c28c74943640d61fdae6fc60613abbf3eb8082a7"} err="failed to get container status \"096bdd50d9cdf384afa596e1c28c74943640d61fdae6fc60613abbf3eb8082a7\": rpc error: code = NotFound desc = could not find container \"096bdd50d9cdf384afa596e1c28c74943640d61fdae6fc60613abbf3eb8082a7\": container with ID starting with 096bdd50d9cdf384afa596e1c28c74943640d61fdae6fc60613abbf3eb8082a7 not found: ID does not exist"
Oct 30 20:30:38.490197 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.490146    5958 scope.go:122] "RemoveContainer" containerID="5804a85bb63e293e5e14f36db543f910ccec895a02f22cc2da98da5a2fcc5d16"
Oct 30 20:30:38.490422 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:30:38.490396    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5804a85bb63e293e5e14f36db543f910ccec895a02f22cc2da98da5a2fcc5d16\": container with ID starting with 5804a85bb63e293e5e14f36db543f910ccec895a02f22cc2da98da5a2fcc5d16 not found: ID does not exist" containerID="5804a85bb63e293e5e14f36db543f910ccec895a02f22cc2da98da5a2fcc5d16"
Oct 30 20:30:38.490530 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.490428    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5804a85bb63e293e5e14f36db543f910ccec895a02f22cc2da98da5a2fcc5d16"} err="failed to get container status \"5804a85bb63e293e5e14f36db543f910ccec895a02f22cc2da98da5a2fcc5d16\": rpc error: code = NotFound desc = could not find container \"5804a85bb63e293e5e14f36db543f910ccec895a02f22cc2da98da5a2fcc5d16\": container with ID starting with 5804a85bb63e293e5e14f36db543f910ccec895a02f22cc2da98da5a2fcc5d16 not found: ID does not exist"
Oct 30 20:30:38.637283 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.637210    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"]
Oct 30 20:30:38.643548 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:38.643517    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-407632-openshift-default-5f64f9cbc5-5kq2v"]
Oct 30 20:30:39.397608 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:39.397579    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fde4d62b-bd67-4915-a145-e5511e651c25" path="/var/lib/kubelet/pods/fde4d62b-bd67-4915-a145-e5511e651c25/volumes"
Oct 30 20:30:42.683852 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:42.683818    5958 scope.go:122] "RemoveContainer" containerID="8d9a4279936677bab550c1ed60930c75a4df66c4bf5722bce068d675519732f9"
Oct 30 20:30:42.692790 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:42.692770    5958 scope.go:122] "RemoveContainer" containerID="acb5f2f01397d1cd306f4e388314cd172aa87e90109bf716db761421670363a6"
Oct 30 20:30:43.230117 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:43.230051    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf" podUID="3e4039e9-32bf-4575-aba9-db44bb0c5077" containerName="payload-processing" containerID="cri-o://4a0db6334aad8dd2b8fe81e42402eba2e7c74ba78edf66b0fb6e03ef3aeb9894" gracePeriod=25
Oct 30 20:30:43.487278 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:43.487221    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf"
Oct 30 20:30:43.499340 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:43.499313    5958 generic.go:356] "Generic (PLEG): container finished" podID="3e4039e9-32bf-4575-aba9-db44bb0c5077" containerID="4a0db6334aad8dd2b8fe81e42402eba2e7c74ba78edf66b0fb6e03ef3aeb9894" exitCode=0
Oct 30 20:30:43.499475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:43.499365    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf"
Oct 30 20:30:43.499475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:43.499395    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf" event={"ID":"3e4039e9-32bf-4575-aba9-db44bb0c5077","Type":"ContainerDied","Data":"4a0db6334aad8dd2b8fe81e42402eba2e7c74ba78edf66b0fb6e03ef3aeb9894"}
Oct 30 20:30:43.499475 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:43.499442    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf" event={"ID":"3e4039e9-32bf-4575-aba9-db44bb0c5077","Type":"ContainerDied","Data":"279bd80050491ee4309d165e5733c68ef644d579543c3945d2a555bb85616f24"}
Oct 30 20:30:43.499599 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:43.499478    5958 scope.go:122] "RemoveContainer" containerID="4a0db6334aad8dd2b8fe81e42402eba2e7c74ba78edf66b0fb6e03ef3aeb9894"
Oct 30 20:30:43.508517 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:43.508498    5958 scope.go:122] "RemoveContainer" containerID="4a0db6334aad8dd2b8fe81e42402eba2e7c74ba78edf66b0fb6e03ef3aeb9894"
Oct 30 20:30:43.508767 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:30:43.508749    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a0db6334aad8dd2b8fe81e42402eba2e7c74ba78edf66b0fb6e03ef3aeb9894\": container with ID starting with 4a0db6334aad8dd2b8fe81e42402eba2e7c74ba78edf66b0fb6e03ef3aeb9894 not found: ID does not exist" containerID="4a0db6334aad8dd2b8fe81e42402eba2e7c74ba78edf66b0fb6e03ef3aeb9894"
Oct 30 20:30:43.508825 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:43.508774    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a0db6334aad8dd2b8fe81e42402eba2e7c74ba78edf66b0fb6e03ef3aeb9894"} err="failed to get container status \"4a0db6334aad8dd2b8fe81e42402eba2e7c74ba78edf66b0fb6e03ef3aeb9894\": rpc error: code = NotFound desc = could not find container \"4a0db6334aad8dd2b8fe81e42402eba2e7c74ba78edf66b0fb6e03ef3aeb9894\": container with ID starting with 4a0db6334aad8dd2b8fe81e42402eba2e7c74ba78edf66b0fb6e03ef3aeb9894 not found: ID does not exist"
Oct 30 20:30:43.569403 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:43.569375    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e4039e9-32bf-4575-aba9-db44bb0c5077-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e4039e9-32bf-4575-aba9-db44bb0c5077-plugins-config-volume\") pod \"3e4039e9-32bf-4575-aba9-db44bb0c5077\" (UID: \"3e4039e9-32bf-4575-aba9-db44bb0c5077\") "
Oct 30 20:30:43.569539 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:43.569433    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e4039e9-32bf-4575-aba9-db44bb0c5077-kube-api-access-g4p55\" (UniqueName: \"kubernetes.io/projected/3e4039e9-32bf-4575-aba9-db44bb0c5077-kube-api-access-g4p55\") pod \"3e4039e9-32bf-4575-aba9-db44bb0c5077\" (UID: \"3e4039e9-32bf-4575-aba9-db44bb0c5077\") "
Oct 30 20:30:43.569770 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:43.569748    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e4039e9-32bf-4575-aba9-db44bb0c5077-plugins-config-volume" pod "3e4039e9-32bf-4575-aba9-db44bb0c5077" (UID: "3e4039e9-32bf-4575-aba9-db44bb0c5077"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:43.569849 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:43.569830    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e4039e9-32bf-4575-aba9-db44bb0c5077-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:43.571594 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:43.571569    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e4039e9-32bf-4575-aba9-db44bb0c5077-kube-api-access-g4p55" pod "3e4039e9-32bf-4575-aba9-db44bb0c5077" (UID: "3e4039e9-32bf-4575-aba9-db44bb0c5077"). InnerVolumeSpecName "kube-api-access-g4p55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:43.671267 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:43.671238    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4p55\" (UniqueName: \"kubernetes.io/projected/3e4039e9-32bf-4575-aba9-db44bb0c5077-kube-api-access-g4p55\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:43.932587 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:43.932552    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf"]
Oct 30 20:30:43.941344 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:43.941313    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-72db69-598798d6b6-jp2zf"]
Oct 30 20:30:45.397161 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:45.397124    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e4039e9-32bf-4575-aba9-db44bb0c5077" path="/var/lib/kubelet/pods/3e4039e9-32bf-4575-aba9-db44bb0c5077/volumes"
Oct 30 20:30:46.240250 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:46.240211    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"]
Oct 30 20:30:46.240465 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:46.240423    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm" podUID="922b8b03-c82e-4d30-9de5-e5d35fe25886" containerName="istio-proxy" containerID="cri-o://49bedcad7053981bfeba7feaf7ca48473539c7a890b3a5896b6b9e24125e0a60" gracePeriod=30
Oct 30 20:30:50.663597 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:50.663518    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72db69-h7knh"]
Oct 30 20:30:50.713136 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:50.713103    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72db69-h7knh"]
Oct 30 20:30:51.398700 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.398658    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3671222c-4562-4e2d-ab26-39aeff8b0c6f" path="/var/lib/kubelet/pods/3671222c-4562-4e2d-ab26-39aeff8b0c6f/volumes"
Oct 30 20:30:51.488627 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.488602    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 20:30:51.546501 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.546418    5958 generic.go:356] "Generic (PLEG): container finished" podID="922b8b03-c82e-4d30-9de5-e5d35fe25886" containerID="49bedcad7053981bfeba7feaf7ca48473539c7a890b3a5896b6b9e24125e0a60" exitCode=0
Oct 30 20:30:51.546501 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.546477    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"
Oct 30 20:30:51.546645 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.546507    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm" event={"ID":"922b8b03-c82e-4d30-9de5-e5d35fe25886","Type":"ContainerDied","Data":"49bedcad7053981bfeba7feaf7ca48473539c7a890b3a5896b6b9e24125e0a60"}
Oct 30 20:30:51.546645 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.546564    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm" event={"ID":"922b8b03-c82e-4d30-9de5-e5d35fe25886","Type":"ContainerDied","Data":"846c0e4e213f48a1870c550c0fdb303acc8a01a0d51e047b2289c2f4832187d1"}
Oct 30 20:30:51.546645 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.546589    5958 scope.go:122] "RemoveContainer" containerID="49bedcad7053981bfeba7feaf7ca48473539c7a890b3a5896b6b9e24125e0a60"
Oct 30 20:30:51.553970 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.553947    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-workload-socket\") pod \"922b8b03-c82e-4d30-9de5-e5d35fe25886\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") "
Oct 30 20:30:51.554095 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.554051    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/922b8b03-c82e-4d30-9de5-e5d35fe25886-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922b8b03-c82e-4d30-9de5-e5d35fe25886-istiod-ca-cert\") pod \"922b8b03-c82e-4d30-9de5-e5d35fe25886\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") "
Oct 30 20:30:51.554095 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.554088    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-credential-socket\") pod \"922b8b03-c82e-4d30-9de5-e5d35fe25886\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") "
Oct 30 20:30:51.554355 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.554182    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-data\") pod \"922b8b03-c82e-4d30-9de5-e5d35fe25886\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") "
Oct 30 20:30:51.554355 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.554210    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-envoy\") pod \"922b8b03-c82e-4d30-9de5-e5d35fe25886\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") "
Oct 30 20:30:51.554355 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.554210    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-workload-socket" pod "922b8b03-c82e-4d30-9de5-e5d35fe25886" (UID: "922b8b03-c82e-4d30-9de5-e5d35fe25886"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:51.554355 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.554246    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-podinfo\") pod \"922b8b03-c82e-4d30-9de5-e5d35fe25886\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") "
Oct 30 20:30:51.554355 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.554293    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922b8b03-c82e-4d30-9de5-e5d35fe25886-kube-api-access-tvsbr\" (UniqueName: \"kubernetes.io/projected/922b8b03-c82e-4d30-9de5-e5d35fe25886-kube-api-access-tvsbr\") pod \"922b8b03-c82e-4d30-9de5-e5d35fe25886\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") "
Oct 30 20:30:51.554651 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.554433    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-credential-socket" pod "922b8b03-c82e-4d30-9de5-e5d35fe25886" (UID: "922b8b03-c82e-4d30-9de5-e5d35fe25886"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:51.554651 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.554478    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/922b8b03-c82e-4d30-9de5-e5d35fe25886-istiod-ca-cert" pod "922b8b03-c82e-4d30-9de5-e5d35fe25886" (UID: "922b8b03-c82e-4d30-9de5-e5d35fe25886"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:30:51.554651 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.554483    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-token\" (UniqueName: \"kubernetes.io/projected/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-token\") pod \"922b8b03-c82e-4d30-9de5-e5d35fe25886\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") "
Oct 30 20:30:51.554651 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.554561    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-workload-certs\") pod \"922b8b03-c82e-4d30-9de5-e5d35fe25886\" (UID: \"922b8b03-c82e-4d30-9de5-e5d35fe25886\") "
Oct 30 20:30:51.554869 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.554839    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-workload-certs" pod "922b8b03-c82e-4d30-9de5-e5d35fe25886" (UID: "922b8b03-c82e-4d30-9de5-e5d35fe25886"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:51.555136 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.554946    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-data" pod "922b8b03-c82e-4d30-9de5-e5d35fe25886" (UID: "922b8b03-c82e-4d30-9de5-e5d35fe25886"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:51.555136 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.555046    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922b8b03-c82e-4d30-9de5-e5d35fe25886-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:51.555136 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.555062    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:51.555136 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.555077    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:51.555136 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.555089    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:51.555136 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.555098    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:51.556039 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.556013    5958 scope.go:122] "RemoveContainer" containerID="49bedcad7053981bfeba7feaf7ca48473539c7a890b3a5896b6b9e24125e0a60"
Oct 30 20:30:51.556419 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:30:51.556394    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49bedcad7053981bfeba7feaf7ca48473539c7a890b3a5896b6b9e24125e0a60\": container with ID starting with 49bedcad7053981bfeba7feaf7ca48473539c7a890b3a5896b6b9e24125e0a60 not found: ID does not exist" containerID="49bedcad7053981bfeba7feaf7ca48473539c7a890b3a5896b6b9e24125e0a60"
Oct 30 20:30:51.556521 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.556433    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49bedcad7053981bfeba7feaf7ca48473539c7a890b3a5896b6b9e24125e0a60"} err="failed to get container status \"49bedcad7053981bfeba7feaf7ca48473539c7a890b3a5896b6b9e24125e0a60\": rpc error: code = NotFound desc = could not find container \"49bedcad7053981bfeba7feaf7ca48473539c7a890b3a5896b6b9e24125e0a60\": container with ID starting with 49bedcad7053981bfeba7feaf7ca48473539c7a890b3a5896b6b9e24125e0a60 not found: ID does not exist"
Oct 30 20:30:51.556769 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.556751    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-podinfo" pod "922b8b03-c82e-4d30-9de5-e5d35fe25886" (UID: "922b8b03-c82e-4d30-9de5-e5d35fe25886"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:30:51.556991 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.556967    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-token" pod "922b8b03-c82e-4d30-9de5-e5d35fe25886" (UID: "922b8b03-c82e-4d30-9de5-e5d35fe25886"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:51.557086 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.557015    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922b8b03-c82e-4d30-9de5-e5d35fe25886-kube-api-access-tvsbr" pod "922b8b03-c82e-4d30-9de5-e5d35fe25886" (UID: "922b8b03-c82e-4d30-9de5-e5d35fe25886"). InnerVolumeSpecName "kube-api-access-tvsbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:30:51.557086 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.557019    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-envoy" pod "922b8b03-c82e-4d30-9de5-e5d35fe25886" (UID: "922b8b03-c82e-4d30-9de5-e5d35fe25886"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:30:51.655803 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.655781    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:51.655803 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.655803    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:51.655930 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.655813    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvsbr\" (UniqueName: \"kubernetes.io/projected/922b8b03-c82e-4d30-9de5-e5d35fe25886-kube-api-access-tvsbr\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:51.655930 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.655822    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922b8b03-c82e-4d30-9de5-e5d35fe25886-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:30:51.967716 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.967674    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"]
Oct 30 20:30:51.971304 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:51.971275    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-lxggm"]
Oct 30 20:30:53.398011 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:53.397971    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="922b8b03-c82e-4d30-9de5-e5d35fe25886" path="/var/lib/kubelet/pods/922b8b03-c82e-4d30-9de5-e5d35fe25886/volumes"
Oct 30 20:30:58.230982 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:58.230948    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"]
Oct 30 20:30:58.231396 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:58.231172    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2" podUID="4b182630-ffc1-4770-a86c-3cdd5b763b92" containerName="istio-proxy" containerID="cri-o://7e238dcda9d4ecb62f1cb474939f2cee3f9f97ce08e435022449d3e7ebcd762f" gracePeriod=30
Oct 30 20:30:59.057412 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.057376    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl"]
Oct 30 20:30:59.057894 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.057875    5958 cpu_manager.go:414] "containerMap: removing container" podUID="32b872e0-6cb5-48ef-9260-5a41099ac864" containerName="cleanup"
Oct 30 20:30:59.057943 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.057896    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b872e0-6cb5-48ef-9260-5a41099ac864" containerName="cleanup"
Oct 30 20:30:59.057943 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.057919    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3671222c-4562-4e2d-ab26-39aeff8b0c6f" containerName="revoke-keys"
Oct 30 20:30:59.057943 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.057924    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3671222c-4562-4e2d-ab26-39aeff8b0c6f" containerName="revoke-keys"
Oct 30 20:30:59.057943 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.057938    5958 cpu_manager.go:414] "containerMap: removing container" podUID="922b8b03-c82e-4d30-9de5-e5d35fe25886" containerName="istio-proxy"
Oct 30 20:30:59.058099 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.057944    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="922b8b03-c82e-4d30-9de5-e5d35fe25886" containerName="istio-proxy"
Oct 30 20:30:59.058099 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.057954    5958 cpu_manager.go:414] "containerMap: removing container" podUID="fde4d62b-bd67-4915-a145-e5511e651c25" containerName="istio-proxy"
Oct 30 20:30:59.058099 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.057961    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde4d62b-bd67-4915-a145-e5511e651c25" containerName="istio-proxy"
Oct 30 20:30:59.058099 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.057988    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3e4039e9-32bf-4575-aba9-db44bb0c5077" containerName="payload-processing"
Oct 30 20:30:59.058099 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.057994    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e4039e9-32bf-4575-aba9-db44bb0c5077" containerName="payload-processing"
Oct 30 20:30:59.058099 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.058008    5958 cpu_manager.go:414] "containerMap: removing container" podUID="fde4d62b-bd67-4915-a145-e5511e651c25" containerName="istio-proxy"
Oct 30 20:30:59.058099 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.058016    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde4d62b-bd67-4915-a145-e5511e651c25" containerName="istio-proxy"
Oct 30 20:30:59.058099 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.058097    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e4039e9-32bf-4575-aba9-db44bb0c5077" containerName="payload-processing"
Oct 30 20:30:59.058331 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.058107    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde4d62b-bd67-4915-a145-e5511e651c25" containerName="istio-proxy"
Oct 30 20:30:59.058331 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.058116    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3671222c-4562-4e2d-ab26-39aeff8b0c6f" containerName="revoke-keys"
Oct 30 20:30:59.058331 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.058122    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="922b8b03-c82e-4d30-9de5-e5d35fe25886" containerName="istio-proxy"
Oct 30 20:30:59.058331 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.058128    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e5a7c3a-bb1a-425e-b0db-934e55ac702f" containerName="cleanup"
Oct 30 20:30:59.062592 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.062575    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl"
Oct 30 20:30:59.116882 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.116854    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:30:59.117407 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.116944    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:30:59.117685 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.117654    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:30:59.117807 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.117776    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:30:59.119638 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.119609    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl"]
Oct 30 20:30:59.232252 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.232219    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0ddc62-7cf6-4f99-8588-25d68e3ec250-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl\" (UID: \"7b0ddc62-7cf6-4f99-8588-25d68e3ec250\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl"
Oct 30 20:30:59.232655 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.232388    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0ddc62-7cf6-4f99-8588-25d68e3ec250-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl\" (UID: \"7b0ddc62-7cf6-4f99-8588-25d68e3ec250\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl"
Oct 30 20:30:59.232655 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.232439    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwh6z\" (UniqueName: \"kubernetes.io/projected/7b0ddc62-7cf6-4f99-8588-25d68e3ec250-kube-api-access-rwh6z\") pod \"maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl\" (UID: \"7b0ddc62-7cf6-4f99-8588-25d68e3ec250\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl"
Oct 30 20:30:59.333617 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.333591    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0ddc62-7cf6-4f99-8588-25d68e3ec250-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl\" (UID: \"7b0ddc62-7cf6-4f99-8588-25d68e3ec250\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl"
Oct 30 20:30:59.333781 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.333638    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwh6z\" (UniqueName: \"kubernetes.io/projected/7b0ddc62-7cf6-4f99-8588-25d68e3ec250-kube-api-access-rwh6z\") pod \"maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl\" (UID: \"7b0ddc62-7cf6-4f99-8588-25d68e3ec250\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl"
Oct 30 20:30:59.333781 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.333669    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0ddc62-7cf6-4f99-8588-25d68e3ec250-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl\" (UID: \"7b0ddc62-7cf6-4f99-8588-25d68e3ec250\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl"
Oct 30 20:30:59.334257 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.334238    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0ddc62-7cf6-4f99-8588-25d68e3ec250-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl\" (UID: \"7b0ddc62-7cf6-4f99-8588-25d68e3ec250\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl"
Oct 30 20:30:59.336069 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.336045    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0ddc62-7cf6-4f99-8588-25d68e3ec250-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl\" (UID: \"7b0ddc62-7cf6-4f99-8588-25d68e3ec250\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl"
Oct 30 20:30:59.396336 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.396311    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwh6z\" (UniqueName: \"kubernetes.io/projected/7b0ddc62-7cf6-4f99-8588-25d68e3ec250-kube-api-access-rwh6z\") pod \"maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl\" (UID: \"7b0ddc62-7cf6-4f99-8588-25d68e3ec250\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl"
Oct 30 20:30:59.675798 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:30:59.675715    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl"
Oct 30 20:31:00.049443 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:31:00.049411    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0ddc62_7cf6_4f99_8588_25d68e3ec250.slice/crio-0944a089c4039b08471978ba26f0dae3c2d1fef2f63700c9972fee5317ce0917 WatchSource:0}: Error finding container 0944a089c4039b08471978ba26f0dae3c2d1fef2f63700c9972fee5317ce0917: Status 404 returned error can't find the container with id 0944a089c4039b08471978ba26f0dae3c2d1fef2f63700c9972fee5317ce0917
Oct 30 20:31:00.094438 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:00.094411    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl"]
Oct 30 20:31:00.592233 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:00.592199    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl" event={"ID":"7b0ddc62-7cf6-4f99-8588-25d68e3ec250","Type":"ContainerStarted","Data":"a352cfd4c46d84ade575c03a11b5b3f98280d9e626dd76ca3342cbbb6731666b"}
Oct 30 20:31:00.592642 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:00.592264    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl" event={"ID":"7b0ddc62-7cf6-4f99-8588-25d68e3ec250","Type":"ContainerStarted","Data":"0944a089c4039b08471978ba26f0dae3c2d1fef2f63700c9972fee5317ce0917"}
Oct 30 20:31:00.709088 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:00.709035    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl" podStartSLOduration=1.709021757 podStartE2EDuration="1.709021757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:00.707876063 +0000 UTC m=+2899.440091455" watchObservedRunningTime="2026-10-30 20:31:00.709021757 +0000 UTC m=+2899.441237147"
Oct 30 20:31:01.598845 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:01.598807    5958 generic.go:356] "Generic (PLEG): container finished" podID="7b0ddc62-7cf6-4f99-8588-25d68e3ec250" containerID="a352cfd4c46d84ade575c03a11b5b3f98280d9e626dd76ca3342cbbb6731666b" exitCode=0
Oct 30 20:31:01.599282 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:01.598890    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl" event={"ID":"7b0ddc62-7cf6-4f99-8588-25d68e3ec250","Type":"ContainerDied","Data":"a352cfd4c46d84ade575c03a11b5b3f98280d9e626dd76ca3342cbbb6731666b"}
Oct 30 20:31:02.749772 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:02.749748    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl"
Oct 30 20:31:02.864912 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:02.864853    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b0ddc62-7cf6-4f99-8588-25d68e3ec250-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0ddc62-7cf6-4f99-8588-25d68e3ec250-maas-api-openshift-ca\") pod \"7b0ddc62-7cf6-4f99-8588-25d68e3ec250\" (UID: \"7b0ddc62-7cf6-4f99-8588-25d68e3ec250\") "
Oct 30 20:31:02.865059 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:02.864921    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b0ddc62-7cf6-4f99-8588-25d68e3ec250-kube-api-access-rwh6z\" (UniqueName: \"kubernetes.io/projected/7b0ddc62-7cf6-4f99-8588-25d68e3ec250-kube-api-access-rwh6z\") pod \"7b0ddc62-7cf6-4f99-8588-25d68e3ec250\" (UID: \"7b0ddc62-7cf6-4f99-8588-25d68e3ec250\") "
Oct 30 20:31:02.865059 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:02.864987    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b0ddc62-7cf6-4f99-8588-25d68e3ec250-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0ddc62-7cf6-4f99-8588-25d68e3ec250-maas-api-xks-ca\") pod \"7b0ddc62-7cf6-4f99-8588-25d68e3ec250\" (UID: \"7b0ddc62-7cf6-4f99-8588-25d68e3ec250\") "
Oct 30 20:31:02.865274 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:02.865239    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b0ddc62-7cf6-4f99-8588-25d68e3ec250-maas-api-openshift-ca" pod "7b0ddc62-7cf6-4f99-8588-25d68e3ec250" (UID: "7b0ddc62-7cf6-4f99-8588-25d68e3ec250"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:02.867401 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:02.867373    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0ddc62-7cf6-4f99-8588-25d68e3ec250-kube-api-access-rwh6z" pod "7b0ddc62-7cf6-4f99-8588-25d68e3ec250" (UID: "7b0ddc62-7cf6-4f99-8588-25d68e3ec250"). InnerVolumeSpecName "kube-api-access-rwh6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:02.867588 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:02.867564    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b0ddc62-7cf6-4f99-8588-25d68e3ec250-maas-api-xks-ca" pod "7b0ddc62-7cf6-4f99-8588-25d68e3ec250" (UID: "7b0ddc62-7cf6-4f99-8588-25d68e3ec250"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:31:02.966433 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:02.966400    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0ddc62-7cf6-4f99-8588-25d68e3ec250-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:02.966433 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:02.966428    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwh6z\" (UniqueName: \"kubernetes.io/projected/7b0ddc62-7cf6-4f99-8588-25d68e3ec250-kube-api-access-rwh6z\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:02.966433 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:02.966438    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0ddc62-7cf6-4f99-8588-25d68e3ec250-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:03.478731 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.478706    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:31:03.619572 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.619542    5958 generic.go:356] "Generic (PLEG): container finished" podID="4b182630-ffc1-4770-a86c-3cdd5b763b92" containerID="7e238dcda9d4ecb62f1cb474939f2cee3f9f97ce08e435022449d3e7ebcd762f" exitCode=0
Oct 30 20:31:03.619715 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.619590    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"
Oct 30 20:31:03.619715 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.619616    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2" event={"ID":"4b182630-ffc1-4770-a86c-3cdd5b763b92","Type":"ContainerDied","Data":"7e238dcda9d4ecb62f1cb474939f2cee3f9f97ce08e435022449d3e7ebcd762f"}
Oct 30 20:31:03.619715 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.619671    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2" event={"ID":"4b182630-ffc1-4770-a86c-3cdd5b763b92","Type":"ContainerDied","Data":"6c928e69b1c09080090a9ef67c6f4699d8532e5ea2f19daf54a1d026071c70b1"}
Oct 30 20:31:03.619715 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.619693    5958 scope.go:122] "RemoveContainer" containerID="7e238dcda9d4ecb62f1cb474939f2cee3f9f97ce08e435022449d3e7ebcd762f"
Oct 30 20:31:03.620986 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.620834    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl"
Oct 30 20:31:03.620986 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.620885    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl" event={"ID":"7b0ddc62-7cf6-4f99-8588-25d68e3ec250","Type":"ContainerDied","Data":"0944a089c4039b08471978ba26f0dae3c2d1fef2f63700c9972fee5317ce0917"}
Oct 30 20:31:03.620986 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.620920    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0944a089c4039b08471978ba26f0dae3c2d1fef2f63700c9972fee5317ce0917"
Oct 30 20:31:03.628552 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.628531    5958 scope.go:122] "RemoveContainer" containerID="fedb85bd1e7a0e8b19c9937cf3e9dc15567d32e16b9bac3ea89b765d2a36d329"
Oct 30 20:31:03.638612 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.638590    5958 scope.go:122] "RemoveContainer" containerID="7e238dcda9d4ecb62f1cb474939f2cee3f9f97ce08e435022449d3e7ebcd762f"
Oct 30 20:31:03.638905 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:31:03.638883    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e238dcda9d4ecb62f1cb474939f2cee3f9f97ce08e435022449d3e7ebcd762f\": container with ID starting with 7e238dcda9d4ecb62f1cb474939f2cee3f9f97ce08e435022449d3e7ebcd762f not found: ID does not exist" containerID="7e238dcda9d4ecb62f1cb474939f2cee3f9f97ce08e435022449d3e7ebcd762f"
Oct 30 20:31:03.638964 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.638915    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e238dcda9d4ecb62f1cb474939f2cee3f9f97ce08e435022449d3e7ebcd762f"} err="failed to get container status \"7e238dcda9d4ecb62f1cb474939f2cee3f9f97ce08e435022449d3e7ebcd762f\": rpc error: code = NotFound desc = could not find container \"7e238dcda9d4ecb62f1cb474939f2cee3f9f97ce08e435022449d3e7ebcd762f\": container with ID starting with 7e238dcda9d4ecb62f1cb474939f2cee3f9f97ce08e435022449d3e7ebcd762f not found: ID does not exist"
Oct 30 20:31:03.638964 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.638936    5958 scope.go:122] "RemoveContainer" containerID="fedb85bd1e7a0e8b19c9937cf3e9dc15567d32e16b9bac3ea89b765d2a36d329"
Oct 30 20:31:03.639227 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:31:03.639208    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fedb85bd1e7a0e8b19c9937cf3e9dc15567d32e16b9bac3ea89b765d2a36d329\": container with ID starting with fedb85bd1e7a0e8b19c9937cf3e9dc15567d32e16b9bac3ea89b765d2a36d329 not found: ID does not exist" containerID="fedb85bd1e7a0e8b19c9937cf3e9dc15567d32e16b9bac3ea89b765d2a36d329"
Oct 30 20:31:03.639295 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.639235    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fedb85bd1e7a0e8b19c9937cf3e9dc15567d32e16b9bac3ea89b765d2a36d329"} err="failed to get container status \"fedb85bd1e7a0e8b19c9937cf3e9dc15567d32e16b9bac3ea89b765d2a36d329\": rpc error: code = NotFound desc = could not find container \"fedb85bd1e7a0e8b19c9937cf3e9dc15567d32e16b9bac3ea89b765d2a36d329\": container with ID starting with fedb85bd1e7a0e8b19c9937cf3e9dc15567d32e16b9bac3ea89b765d2a36d329 not found: ID does not exist"
Oct 30 20:31:03.672567 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.672542    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-envoy\") pod \"4b182630-ffc1-4770-a86c-3cdd5b763b92\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") "
Oct 30 20:31:03.672691 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.672575    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b182630-ffc1-4770-a86c-3cdd5b763b92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b182630-ffc1-4770-a86c-3cdd5b763b92-istiod-ca-cert\") pod \"4b182630-ffc1-4770-a86c-3cdd5b763b92\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") "
Oct 30 20:31:03.672691 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.672609    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-token\" (UniqueName: \"kubernetes.io/projected/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-token\") pod \"4b182630-ffc1-4770-a86c-3cdd5b763b92\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") "
Oct 30 20:31:03.672691 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.672637    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-data\") pod \"4b182630-ffc1-4770-a86c-3cdd5b763b92\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") "
Oct 30 20:31:03.672691 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.672671    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b182630-ffc1-4770-a86c-3cdd5b763b92-kube-api-access-qd52l\" (UniqueName: \"kubernetes.io/projected/4b182630-ffc1-4770-a86c-3cdd5b763b92-kube-api-access-qd52l\") pod \"4b182630-ffc1-4770-a86c-3cdd5b763b92\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") "
Oct 30 20:31:03.672913 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.672756    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-podinfo\") pod \"4b182630-ffc1-4770-a86c-3cdd5b763b92\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") "
Oct 30 20:31:03.672913 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.672811    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-credential-socket\") pod \"4b182630-ffc1-4770-a86c-3cdd5b763b92\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") "
Oct 30 20:31:03.672913 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.672884    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-workload-certs\") pod \"4b182630-ffc1-4770-a86c-3cdd5b763b92\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") "
Oct 30 20:31:03.673068 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.672986    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b182630-ffc1-4770-a86c-3cdd5b763b92-istiod-ca-cert" pod "4b182630-ffc1-4770-a86c-3cdd5b763b92" (UID: "4b182630-ffc1-4770-a86c-3cdd5b763b92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:03.673068 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.673000    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-workload-socket\") pod \"4b182630-ffc1-4770-a86c-3cdd5b763b92\" (UID: \"4b182630-ffc1-4770-a86c-3cdd5b763b92\") "
Oct 30 20:31:03.673200 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.673174    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-data" pod "4b182630-ffc1-4770-a86c-3cdd5b763b92" (UID: "4b182630-ffc1-4770-a86c-3cdd5b763b92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:03.673403 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.673378    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:03.673499 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.673410    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b182630-ffc1-4770-a86c-3cdd5b763b92-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:03.673613 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.673589    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-workload-socket" pod "4b182630-ffc1-4770-a86c-3cdd5b763b92" (UID: "4b182630-ffc1-4770-a86c-3cdd5b763b92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:03.673710 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.673688    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-credential-socket" pod "4b182630-ffc1-4770-a86c-3cdd5b763b92" (UID: "4b182630-ffc1-4770-a86c-3cdd5b763b92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:03.673823 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.673798    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-workload-certs" pod "4b182630-ffc1-4770-a86c-3cdd5b763b92" (UID: "4b182630-ffc1-4770-a86c-3cdd5b763b92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:03.675637 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.675606    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-podinfo" pod "4b182630-ffc1-4770-a86c-3cdd5b763b92" (UID: "4b182630-ffc1-4770-a86c-3cdd5b763b92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:31:03.675771 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.675739    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-token" pod "4b182630-ffc1-4770-a86c-3cdd5b763b92" (UID: "4b182630-ffc1-4770-a86c-3cdd5b763b92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:03.675892 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.675799    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-envoy" pod "4b182630-ffc1-4770-a86c-3cdd5b763b92" (UID: "4b182630-ffc1-4770-a86c-3cdd5b763b92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:03.676062 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.676044    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b182630-ffc1-4770-a86c-3cdd5b763b92-kube-api-access-qd52l" pod "4b182630-ffc1-4770-a86c-3cdd5b763b92" (UID: "4b182630-ffc1-4770-a86c-3cdd5b763b92"). InnerVolumeSpecName "kube-api-access-qd52l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:03.744709 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.744673    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"]
Oct 30 20:31:03.774522 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.774494    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:03.774522 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.774522    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:03.774872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.774533    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd52l\" (UniqueName: \"kubernetes.io/projected/4b182630-ffc1-4770-a86c-3cdd5b763b92-kube-api-access-qd52l\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:03.774872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.774544    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b182630-ffc1-4770-a86c-3cdd5b763b92-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:03.774872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.774553    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:03.774872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.774562    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:03.774872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:03.774570    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b182630-ffc1-4770-a86c-3cdd5b763b92-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:04.041252 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:04.041221    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"]
Oct 30 20:31:04.044952 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:04.044930    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-72db69-openshift-default-5dfb4b6d65-dcxh2"]
Oct 30 20:31:05.398175 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:05.398144    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b182630-ffc1-4770-a86c-3cdd5b763b92" path="/var/lib/kubelet/pods/4b182630-ffc1-4770-a86c-3cdd5b763b92/volumes"
Oct 30 20:31:06.605912 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:31:06.605876    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:31:09.817074 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:09.817040    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b954773c-gjrcj"]
Oct 30 20:31:09.817498 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:09.817485    5958 cpu_manager.go:414] "containerMap: removing container" podUID="7b0ddc62-7cf6-4f99-8588-25d68e3ec250" containerName="revoke-keys"
Oct 30 20:31:09.817566 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:09.817499    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b0ddc62-7cf6-4f99-8588-25d68e3ec250" containerName="revoke-keys"
Oct 30 20:31:09.817566 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:09.817508    5958 cpu_manager.go:414] "containerMap: removing container" podUID="4b182630-ffc1-4770-a86c-3cdd5b763b92" containerName="istio-proxy"
Oct 30 20:31:09.817566 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:09.817513    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b182630-ffc1-4770-a86c-3cdd5b763b92" containerName="istio-proxy"
Oct 30 20:31:09.817661 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:09.817591    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b0ddc62-7cf6-4f99-8588-25d68e3ec250" containerName="revoke-keys"
Oct 30 20:31:09.817661 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:09.817600    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b182630-ffc1-4770-a86c-3cdd5b763b92" containerName="istio-proxy"
Oct 30 20:31:09.817661 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:09.817608    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b182630-ffc1-4770-a86c-3cdd5b763b92" containerName="istio-proxy"
Oct 30 20:31:09.817661 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:09.817615    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde4d62b-bd67-4915-a145-e5511e651c25" containerName="istio-proxy"
Oct 30 20:31:09.820721 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:09.820697    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b954773c-gjrcj"
Oct 30 20:31:09.824053 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:09.824025    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b96a18b-6691-4463-88dd-933e4057f7b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b954773c-gjrcj\" (UID: \"3b96a18b-6691-4463-88dd-933e4057f7b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b954773c-gjrcj"
Oct 30 20:31:09.824163 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:09.824081    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b96a18b-6691-4463-88dd-933e4057f7b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b954773c-gjrcj\" (UID: \"3b96a18b-6691-4463-88dd-933e4057f7b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b954773c-gjrcj"
Oct 30 20:31:09.824227 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:09.824195    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsrwp\" (UniqueName: \"kubernetes.io/projected/3b96a18b-6691-4463-88dd-933e4057f7b4-kube-api-access-qsrwp\") pod \"maas-api-revoke-sub-b954773c-gjrcj\" (UID: \"3b96a18b-6691-4463-88dd-933e4057f7b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b954773c-gjrcj"
Oct 30 20:31:09.873603 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:09.873583    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:31:09.923318 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:09.923299    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:31:09.923318 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:09.923312    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:31:09.923446 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:09.923299    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:31:09.924609 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:09.924588    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b96a18b-6691-4463-88dd-933e4057f7b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b954773c-gjrcj\" (UID: \"3b96a18b-6691-4463-88dd-933e4057f7b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b954773c-gjrcj"
Oct 30 20:31:09.924713 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:09.924627    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b96a18b-6691-4463-88dd-933e4057f7b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b954773c-gjrcj\" (UID: \"3b96a18b-6691-4463-88dd-933e4057f7b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b954773c-gjrcj"
Oct 30 20:31:09.924713 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:09.924662    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsrwp\" (UniqueName: \"kubernetes.io/projected/3b96a18b-6691-4463-88dd-933e4057f7b4-kube-api-access-qsrwp\") pod \"maas-api-revoke-sub-b954773c-gjrcj\" (UID: \"3b96a18b-6691-4463-88dd-933e4057f7b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b954773c-gjrcj"
Oct 30 20:31:09.925190 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:09.925171    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b96a18b-6691-4463-88dd-933e4057f7b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b954773c-gjrcj\" (UID: \"3b96a18b-6691-4463-88dd-933e4057f7b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b954773c-gjrcj"
Oct 30 20:31:09.927120 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:09.927097    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b96a18b-6691-4463-88dd-933e4057f7b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b954773c-gjrcj\" (UID: \"3b96a18b-6691-4463-88dd-933e4057f7b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b954773c-gjrcj"
Oct 30 20:31:09.932253 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:09.932229    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b954773c-gjrcj"]
Oct 30 20:31:09.986003 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:09.985978    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsrwp\" (UniqueName: \"kubernetes.io/projected/3b96a18b-6691-4463-88dd-933e4057f7b4-kube-api-access-qsrwp\") pod \"maas-api-revoke-sub-b954773c-gjrcj\" (UID: \"3b96a18b-6691-4463-88dd-933e4057f7b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b954773c-gjrcj"
Oct 30 20:31:10.134090 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:10.134031    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b954773c-gjrcj"
Oct 30 20:31:10.505918 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:10.505816    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b954773c-gjrcj"]
Oct 30 20:31:10.508882 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:31:10.508855    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b96a18b_6691_4463_88dd_933e4057f7b4.slice/crio-0834635a6539eac82efc45f6ff0d02ce429c23e9aaa62d478d7bdf2a5e37b6c3 WatchSource:0}: Error finding container 0834635a6539eac82efc45f6ff0d02ce429c23e9aaa62d478d7bdf2a5e37b6c3: Status 404 returned error can't find the container with id 0834635a6539eac82efc45f6ff0d02ce429c23e9aaa62d478d7bdf2a5e37b6c3
Oct 30 20:31:10.657284 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:10.657250    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b954773c-gjrcj" event={"ID":"3b96a18b-6691-4463-88dd-933e4057f7b4","Type":"ContainerStarted","Data":"0834635a6539eac82efc45f6ff0d02ce429c23e9aaa62d478d7bdf2a5e37b6c3"}
Oct 30 20:31:10.767413 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:10.767338    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv"]
Oct 30 20:31:11.664315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:11.664242    5958 generic.go:356] "Generic (PLEG): container finished" podID="3b96a18b-6691-4463-88dd-933e4057f7b4" containerID="d7279136a8f7e65923a7b127bd8500fd89cef80336eabec703698836efd971c6" exitCode=0
Oct 30 20:31:11.664663 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:11.664334    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b954773c-gjrcj" event={"ID":"3b96a18b-6691-4463-88dd-933e4057f7b4","Type":"ContainerDied","Data":"d7279136a8f7e65923a7b127bd8500fd89cef80336eabec703698836efd971c6"}
Oct 30 20:31:12.803672 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:12.803646    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b954773c-gjrcj"
Oct 30 20:31:12.851005 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:12.850980    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b96a18b-6691-4463-88dd-933e4057f7b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b96a18b-6691-4463-88dd-933e4057f7b4-maas-api-openshift-ca\") pod \"3b96a18b-6691-4463-88dd-933e4057f7b4\" (UID: \"3b96a18b-6691-4463-88dd-933e4057f7b4\") "
Oct 30 20:31:12.851105 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:12.851092    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b96a18b-6691-4463-88dd-933e4057f7b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b96a18b-6691-4463-88dd-933e4057f7b4-maas-api-xks-ca\") pod \"3b96a18b-6691-4463-88dd-933e4057f7b4\" (UID: \"3b96a18b-6691-4463-88dd-933e4057f7b4\") "
Oct 30 20:31:12.851142 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:12.851136    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b96a18b-6691-4463-88dd-933e4057f7b4-kube-api-access-qsrwp\" (UniqueName: \"kubernetes.io/projected/3b96a18b-6691-4463-88dd-933e4057f7b4-kube-api-access-qsrwp\") pod \"3b96a18b-6691-4463-88dd-933e4057f7b4\" (UID: \"3b96a18b-6691-4463-88dd-933e4057f7b4\") "
Oct 30 20:31:12.851316 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:12.851296    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b96a18b-6691-4463-88dd-933e4057f7b4-maas-api-openshift-ca" pod "3b96a18b-6691-4463-88dd-933e4057f7b4" (UID: "3b96a18b-6691-4463-88dd-933e4057f7b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:12.851521 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:12.851499    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b96a18b-6691-4463-88dd-933e4057f7b4-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:12.853244 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:12.853220    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b96a18b-6691-4463-88dd-933e4057f7b4-maas-api-xks-ca" pod "3b96a18b-6691-4463-88dd-933e4057f7b4" (UID: "3b96a18b-6691-4463-88dd-933e4057f7b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:31:12.853338 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:12.853283    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b96a18b-6691-4463-88dd-933e4057f7b4-kube-api-access-qsrwp" pod "3b96a18b-6691-4463-88dd-933e4057f7b4" (UID: "3b96a18b-6691-4463-88dd-933e4057f7b4"). InnerVolumeSpecName "kube-api-access-qsrwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:12.952963 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:12.952901    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsrwp\" (UniqueName: \"kubernetes.io/projected/3b96a18b-6691-4463-88dd-933e4057f7b4-kube-api-access-qsrwp\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:12.952963 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:12.952928    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b96a18b-6691-4463-88dd-933e4057f7b4-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:13.675002 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:13.674929    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b954773c-gjrcj"
Oct 30 20:31:13.675002 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:13.674939    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b954773c-gjrcj" event={"ID":"3b96a18b-6691-4463-88dd-933e4057f7b4","Type":"ContainerDied","Data":"0834635a6539eac82efc45f6ff0d02ce429c23e9aaa62d478d7bdf2a5e37b6c3"}
Oct 30 20:31:13.675236 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:13.675007    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0834635a6539eac82efc45f6ff0d02ce429c23e9aaa62d478d7bdf2a5e37b6c3"
Oct 30 20:31:15.767867 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:15.767830    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv" podUID="a984e6de-54fc-4ec0-83b0-ab295078699b" containerName="payload-processing" containerID="cri-o://5ea23c4035783d7c6ef73fb410cad68711dcfc8f71babfd04ff0a24c814a3020" gracePeriod=25
Oct 30 20:31:16.016436 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:16.016413    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv"
Oct 30 20:31:16.077973 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:16.077949    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a984e6de-54fc-4ec0-83b0-ab295078699b-kube-api-access-7nfpb\" (UniqueName: \"kubernetes.io/projected/a984e6de-54fc-4ec0-83b0-ab295078699b-kube-api-access-7nfpb\") pod \"a984e6de-54fc-4ec0-83b0-ab295078699b\" (UID: \"a984e6de-54fc-4ec0-83b0-ab295078699b\") "
Oct 30 20:31:16.078101 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:16.078088    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a984e6de-54fc-4ec0-83b0-ab295078699b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a984e6de-54fc-4ec0-83b0-ab295078699b-plugins-config-volume\") pod \"a984e6de-54fc-4ec0-83b0-ab295078699b\" (UID: \"a984e6de-54fc-4ec0-83b0-ab295078699b\") "
Oct 30 20:31:16.078514 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:16.078488    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a984e6de-54fc-4ec0-83b0-ab295078699b-plugins-config-volume" pod "a984e6de-54fc-4ec0-83b0-ab295078699b" (UID: "a984e6de-54fc-4ec0-83b0-ab295078699b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:16.080002 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:16.079981    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a984e6de-54fc-4ec0-83b0-ab295078699b-kube-api-access-7nfpb" pod "a984e6de-54fc-4ec0-83b0-ab295078699b" (UID: "a984e6de-54fc-4ec0-83b0-ab295078699b"). InnerVolumeSpecName "kube-api-access-7nfpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:16.178760 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:16.178736    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nfpb\" (UniqueName: \"kubernetes.io/projected/a984e6de-54fc-4ec0-83b0-ab295078699b-kube-api-access-7nfpb\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:16.178760 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:16.178761    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a984e6de-54fc-4ec0-83b0-ab295078699b-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:16.692656 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:16.692630    5958 generic.go:356] "Generic (PLEG): container finished" podID="a984e6de-54fc-4ec0-83b0-ab295078699b" containerID="5ea23c4035783d7c6ef73fb410cad68711dcfc8f71babfd04ff0a24c814a3020" exitCode=0
Oct 30 20:31:16.692817 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:16.692675    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv"
Oct 30 20:31:16.692817 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:16.692725    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv" event={"ID":"a984e6de-54fc-4ec0-83b0-ab295078699b","Type":"ContainerDied","Data":"5ea23c4035783d7c6ef73fb410cad68711dcfc8f71babfd04ff0a24c814a3020"}
Oct 30 20:31:16.692817 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:16.692775    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv" event={"ID":"a984e6de-54fc-4ec0-83b0-ab295078699b","Type":"ContainerDied","Data":"efd5881e89499709f6e1edaea465de58f5f3ccb73105b0309fa4a1548be7f3c5"}
Oct 30 20:31:16.692817 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:16.692797    5958 scope.go:122] "RemoveContainer" containerID="5ea23c4035783d7c6ef73fb410cad68711dcfc8f71babfd04ff0a24c814a3020"
Oct 30 20:31:16.702419 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:16.702398    5958 scope.go:122] "RemoveContainer" containerID="5ea23c4035783d7c6ef73fb410cad68711dcfc8f71babfd04ff0a24c814a3020"
Oct 30 20:31:16.702719 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:31:16.702695    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ea23c4035783d7c6ef73fb410cad68711dcfc8f71babfd04ff0a24c814a3020\": container with ID starting with 5ea23c4035783d7c6ef73fb410cad68711dcfc8f71babfd04ff0a24c814a3020 not found: ID does not exist" containerID="5ea23c4035783d7c6ef73fb410cad68711dcfc8f71babfd04ff0a24c814a3020"
Oct 30 20:31:16.702773 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:16.702731    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ea23c4035783d7c6ef73fb410cad68711dcfc8f71babfd04ff0a24c814a3020"} err="failed to get container status \"5ea23c4035783d7c6ef73fb410cad68711dcfc8f71babfd04ff0a24c814a3020\": rpc error: code = NotFound desc = could not find container \"5ea23c4035783d7c6ef73fb410cad68711dcfc8f71babfd04ff0a24c814a3020\": container with ID starting with 5ea23c4035783d7c6ef73fb410cad68711dcfc8f71babfd04ff0a24c814a3020 not found: ID does not exist"
Oct 30 20:31:16.856677 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:16.856647    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv"]
Oct 30 20:31:16.861221 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:16.861199    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-3ccc9f-7b58b44bf4-2nhrv"]
Oct 30 20:31:17.396843 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:17.396812    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a984e6de-54fc-4ec0-83b0-ab295078699b" path="/var/lib/kubelet/pods/a984e6de-54fc-4ec0-83b0-ab295078699b/volumes"
Oct 30 20:31:18.765766 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:18.765722    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main" containerID="cri-o://788f9157e90aaf7b69db4bc12c41735e449a0be4da9d466830e776acb8138453" gracePeriod=45
Oct 30 20:31:19.008511 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.008462    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:31:19.103980 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.103955    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-model-cache\") pod \"504cc4c3-8577-43e6-8feb-5263e0d7d310\" (UID: \"504cc4c3-8577-43e6-8feb-5263e0d7d310\") "
Oct 30 20:31:19.104089 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.103990    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-home\" (UniqueName: \"kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-home\") pod \"504cc4c3-8577-43e6-8feb-5263e0d7d310\" (UID: \"504cc4c3-8577-43e6-8feb-5263e0d7d310\") "
Oct 30 20:31:19.104089 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.104013    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/504cc4c3-8577-43e6-8feb-5263e0d7d310-kube-api-access-mxssw\" (UniqueName: \"kubernetes.io/projected/504cc4c3-8577-43e6-8feb-5263e0d7d310-kube-api-access-mxssw\") pod \"504cc4c3-8577-43e6-8feb-5263e0d7d310\" (UID: \"504cc4c3-8577-43e6-8feb-5263e0d7d310\") "
Oct 30 20:31:19.104172 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.104109    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/504cc4c3-8577-43e6-8feb-5263e0d7d310-tls-certs\" (UniqueName: \"kubernetes.io/secret/504cc4c3-8577-43e6-8feb-5263e0d7d310-tls-certs\") pod \"504cc4c3-8577-43e6-8feb-5263e0d7d310\" (UID: \"504cc4c3-8577-43e6-8feb-5263e0d7d310\") "
Oct 30 20:31:19.104172 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.104149    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-dshm\" (UniqueName: \"kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-dshm\") pod \"504cc4c3-8577-43e6-8feb-5263e0d7d310\" (UID: \"504cc4c3-8577-43e6-8feb-5263e0d7d310\") "
Oct 30 20:31:19.104273 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.104181    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-tmp-dir\") pod \"504cc4c3-8577-43e6-8feb-5263e0d7d310\" (UID: \"504cc4c3-8577-43e6-8feb-5263e0d7d310\") "
Oct 30 20:31:19.104273 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.104223    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-home" pod "504cc4c3-8577-43e6-8feb-5263e0d7d310" (UID: "504cc4c3-8577-43e6-8feb-5263e0d7d310"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:19.104273 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.104250    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-model-cache" pod "504cc4c3-8577-43e6-8feb-5263e0d7d310" (UID: "504cc4c3-8577-43e6-8feb-5263e0d7d310"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:19.104550 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.104527    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-tmp-dir" pod "504cc4c3-8577-43e6-8feb-5263e0d7d310" (UID: "504cc4c3-8577-43e6-8feb-5263e0d7d310"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:19.104550 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.104540    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-model-cache\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:19.104880 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.104560    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-home\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:19.106366 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.106343    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/504cc4c3-8577-43e6-8feb-5263e0d7d310-tls-certs" pod "504cc4c3-8577-43e6-8feb-5263e0d7d310" (UID: "504cc4c3-8577-43e6-8feb-5263e0d7d310"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:31:19.106366 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.106355    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/504cc4c3-8577-43e6-8feb-5263e0d7d310-kube-api-access-mxssw" pod "504cc4c3-8577-43e6-8feb-5263e0d7d310" (UID: "504cc4c3-8577-43e6-8feb-5263e0d7d310"). InnerVolumeSpecName "kube-api-access-mxssw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:19.106366 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.106356    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-dshm" pod "504cc4c3-8577-43e6-8feb-5263e0d7d310" (UID: "504cc4c3-8577-43e6-8feb-5263e0d7d310"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:19.205542 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.205509    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-dshm\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:19.205542 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.205544    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/504cc4c3-8577-43e6-8feb-5263e0d7d310-tmp-dir\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:19.205785 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.205556    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxssw\" (UniqueName: \"kubernetes.io/projected/504cc4c3-8577-43e6-8feb-5263e0d7d310-kube-api-access-mxssw\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:19.205785 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.205564    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/504cc4c3-8577-43e6-8feb-5263e0d7d310-tls-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:19.711736 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.711639    5958 generic.go:356] "Generic (PLEG): container finished" podID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerID="788f9157e90aaf7b69db4bc12c41735e449a0be4da9d466830e776acb8138453" exitCode=0
Oct 30 20:31:19.711736 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.711699    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"
Oct 30 20:31:19.711964 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.711728    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" event={"ID":"504cc4c3-8577-43e6-8feb-5263e0d7d310","Type":"ContainerDied","Data":"788f9157e90aaf7b69db4bc12c41735e449a0be4da9d466830e776acb8138453"}
Oct 30 20:31:19.711964 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.711782    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8" event={"ID":"504cc4c3-8577-43e6-8feb-5263e0d7d310","Type":"ContainerDied","Data":"9b310dab574aa6c5faa4ff86167bcd91e5a15cfbcbce9446880f9ddf20583e2b"}
Oct 30 20:31:19.711964 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.711806    5958 scope.go:122] "RemoveContainer" containerID="788f9157e90aaf7b69db4bc12c41735e449a0be4da9d466830e776acb8138453"
Oct 30 20:31:19.720360 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.720337    5958 scope.go:122] "RemoveContainer" containerID="788f9157e90aaf7b69db4bc12c41735e449a0be4da9d466830e776acb8138453"
Oct 30 20:31:19.720663 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:31:19.720643    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"788f9157e90aaf7b69db4bc12c41735e449a0be4da9d466830e776acb8138453\": container with ID starting with 788f9157e90aaf7b69db4bc12c41735e449a0be4da9d466830e776acb8138453 not found: ID does not exist" containerID="788f9157e90aaf7b69db4bc12c41735e449a0be4da9d466830e776acb8138453"
Oct 30 20:31:19.720727 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.720673    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"788f9157e90aaf7b69db4bc12c41735e449a0be4da9d466830e776acb8138453"} err="failed to get container status \"788f9157e90aaf7b69db4bc12c41735e449a0be4da9d466830e776acb8138453\": rpc error: code = NotFound desc = could not find container \"788f9157e90aaf7b69db4bc12c41735e449a0be4da9d466830e776acb8138453\": container with ID starting with 788f9157e90aaf7b69db4bc12c41735e449a0be4da9d466830e776acb8138453 not found: ID does not exist"
Oct 30 20:31:19.876997 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.876970    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"]
Oct 30 20:31:19.926872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:19.926847    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-3ccc9f/distinct-w1-3ccc9f-kserve-7b6b667c75-69hq8"]
Oct 30 20:31:21.397460 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:21.397413    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" path="/var/lib/kubelet/pods/504cc4c3-8577-43e6-8feb-5263e0d7d310/volumes"
Oct 30 20:31:26.643558 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:26.643530    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl"]
Oct 30 20:31:26.693576 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:26.693544    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3ccc9f-9n2zl"]
Oct 30 20:31:27.396391 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:27.396359    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b0ddc62-7cf6-4f99-8588-25d68e3ec250" path="/var/lib/kubelet/pods/7b0ddc62-7cf6-4f99-8588-25d68e3ec250/volumes"
Oct 30 20:31:34.535496 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:34.535444    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"]
Oct 30 20:31:34.535890 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:34.535676    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm" podUID="c707594b-f9af-4025-8614-43ce0feaa1e9" containerName="istio-proxy" containerID="cri-o://18464ae24f428ef0cc5013d5f2a7c9381ec2f15bd3027d1fc49f7bc54a700618" gracePeriod=30
Oct 30 20:31:36.604605 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:31:36.604568    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:31:39.782444 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.782421    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:31:39.813781 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.813750    5958 generic.go:356] "Generic (PLEG): container finished" podID="c707594b-f9af-4025-8614-43ce0feaa1e9" containerID="18464ae24f428ef0cc5013d5f2a7c9381ec2f15bd3027d1fc49f7bc54a700618" exitCode=0
Oct 30 20:31:39.813902 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.813792    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"
Oct 30 20:31:39.813902 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.813834    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm" event={"ID":"c707594b-f9af-4025-8614-43ce0feaa1e9","Type":"ContainerDied","Data":"18464ae24f428ef0cc5013d5f2a7c9381ec2f15bd3027d1fc49f7bc54a700618"}
Oct 30 20:31:39.813902 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.813881    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm" event={"ID":"c707594b-f9af-4025-8614-43ce0feaa1e9","Type":"ContainerDied","Data":"ee72e32027e1c685fca850ba1fe29ff746be2dc9387abf84d688e228914a1fc8"}
Oct 30 20:31:39.813902 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.813901    5958 scope.go:122] "RemoveContainer" containerID="18464ae24f428ef0cc5013d5f2a7c9381ec2f15bd3027d1fc49f7bc54a700618"
Oct 30 20:31:39.822926 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.822907    5958 scope.go:122] "RemoveContainer" containerID="18464ae24f428ef0cc5013d5f2a7c9381ec2f15bd3027d1fc49f7bc54a700618"
Oct 30 20:31:39.823202 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:31:39.823185    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18464ae24f428ef0cc5013d5f2a7c9381ec2f15bd3027d1fc49f7bc54a700618\": container with ID starting with 18464ae24f428ef0cc5013d5f2a7c9381ec2f15bd3027d1fc49f7bc54a700618 not found: ID does not exist" containerID="18464ae24f428ef0cc5013d5f2a7c9381ec2f15bd3027d1fc49f7bc54a700618"
Oct 30 20:31:39.823279 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.823208    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18464ae24f428ef0cc5013d5f2a7c9381ec2f15bd3027d1fc49f7bc54a700618"} err="failed to get container status \"18464ae24f428ef0cc5013d5f2a7c9381ec2f15bd3027d1fc49f7bc54a700618\": rpc error: code = NotFound desc = could not find container \"18464ae24f428ef0cc5013d5f2a7c9381ec2f15bd3027d1fc49f7bc54a700618\": container with ID starting with 18464ae24f428ef0cc5013d5f2a7c9381ec2f15bd3027d1fc49f7bc54a700618 not found: ID does not exist"
Oct 30 20:31:39.878428 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.878406    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-token\" (UniqueName: \"kubernetes.io/projected/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-token\") pod \"c707594b-f9af-4025-8614-43ce0feaa1e9\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") "
Oct 30 20:31:39.878546 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.878481    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-envoy\") pod \"c707594b-f9af-4025-8614-43ce0feaa1e9\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") "
Oct 30 20:31:39.878546 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.878500    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-workload-socket\") pod \"c707594b-f9af-4025-8614-43ce0feaa1e9\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") "
Oct 30 20:31:39.878546 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.878517    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c707594b-f9af-4025-8614-43ce0feaa1e9-kube-api-access-pmbln\" (UniqueName: \"kubernetes.io/projected/c707594b-f9af-4025-8614-43ce0feaa1e9-kube-api-access-pmbln\") pod \"c707594b-f9af-4025-8614-43ce0feaa1e9\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") "
Oct 30 20:31:39.878546 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.878540    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-podinfo\") pod \"c707594b-f9af-4025-8614-43ce0feaa1e9\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") "
Oct 30 20:31:39.878746 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.878584    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-credential-socket\") pod \"c707594b-f9af-4025-8614-43ce0feaa1e9\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") "
Oct 30 20:31:39.878746 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.878654    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-workload-certs\") pod \"c707594b-f9af-4025-8614-43ce0feaa1e9\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") "
Oct 30 20:31:39.878746 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.878688    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c707594b-f9af-4025-8614-43ce0feaa1e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c707594b-f9af-4025-8614-43ce0feaa1e9-istiod-ca-cert\") pod \"c707594b-f9af-4025-8614-43ce0feaa1e9\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") "
Oct 30 20:31:39.878912 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.878766    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-workload-socket" pod "c707594b-f9af-4025-8614-43ce0feaa1e9" (UID: "c707594b-f9af-4025-8614-43ce0feaa1e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:39.878912 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.878782    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-data\") pod \"c707594b-f9af-4025-8614-43ce0feaa1e9\" (UID: \"c707594b-f9af-4025-8614-43ce0feaa1e9\") "
Oct 30 20:31:39.879033 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.878931    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-workload-certs" pod "c707594b-f9af-4025-8614-43ce0feaa1e9" (UID: "c707594b-f9af-4025-8614-43ce0feaa1e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:39.879033 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.878967    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-credential-socket" pod "c707594b-f9af-4025-8614-43ce0feaa1e9" (UID: "c707594b-f9af-4025-8614-43ce0feaa1e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:39.879226 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.879185    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-data" pod "c707594b-f9af-4025-8614-43ce0feaa1e9" (UID: "c707594b-f9af-4025-8614-43ce0feaa1e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:39.879471 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.879414    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:39.879471 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.879437    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:39.879595 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.879471    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:39.879595 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.879486    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:39.879595 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.879570    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c707594b-f9af-4025-8614-43ce0feaa1e9-istiod-ca-cert" pod "c707594b-f9af-4025-8614-43ce0feaa1e9" (UID: "c707594b-f9af-4025-8614-43ce0feaa1e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:39.880832 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.880798    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-podinfo" pod "c707594b-f9af-4025-8614-43ce0feaa1e9" (UID: "c707594b-f9af-4025-8614-43ce0feaa1e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:31:39.880832 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.880821    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-token" pod "c707594b-f9af-4025-8614-43ce0feaa1e9" (UID: "c707594b-f9af-4025-8614-43ce0feaa1e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:39.881204 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.881190    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-envoy" pod "c707594b-f9af-4025-8614-43ce0feaa1e9" (UID: "c707594b-f9af-4025-8614-43ce0feaa1e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:39.881388 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.881368    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c707594b-f9af-4025-8614-43ce0feaa1e9-kube-api-access-pmbln" pod "c707594b-f9af-4025-8614-43ce0feaa1e9" (UID: "c707594b-f9af-4025-8614-43ce0feaa1e9"). InnerVolumeSpecName "kube-api-access-pmbln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:39.980712 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.980687    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:39.980712 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.980709    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:39.980909 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.980719    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmbln\" (UniqueName: \"kubernetes.io/projected/c707594b-f9af-4025-8614-43ce0feaa1e9-kube-api-access-pmbln\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:39.980909 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.980729    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c707594b-f9af-4025-8614-43ce0feaa1e9-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:39.980909 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:39.980738    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c707594b-f9af-4025-8614-43ce0feaa1e9-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:40.235289 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:40.235256    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"]
Oct 30 20:31:40.241771 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:40.241747    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3ccc9f-openshift-default-6869878bbf-4psfm"]
Oct 30 20:31:41.397370 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:31:41.397339    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c707594b-f9af-4025-8614-43ce0feaa1e9" path="/var/lib/kubelet/pods/c707594b-f9af-4025-8614-43ce0feaa1e9/volumes"
Oct 30 20:32:04.207138 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.207102    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nm9rf"]
Oct 30 20:32:04.207642 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.207517    5958 cpu_manager.go:414] "containerMap: removing container" podUID="4b182630-ffc1-4770-a86c-3cdd5b763b92" containerName="istio-proxy"
Oct 30 20:32:04.207642 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.207530    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b182630-ffc1-4770-a86c-3cdd5b763b92" containerName="istio-proxy"
Oct 30 20:32:04.207642 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.207562    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c707594b-f9af-4025-8614-43ce0feaa1e9" containerName="istio-proxy"
Oct 30 20:32:04.207642 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.207567    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c707594b-f9af-4025-8614-43ce0feaa1e9" containerName="istio-proxy"
Oct 30 20:32:04.207642 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.207576    5958 cpu_manager.go:414] "containerMap: removing container" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main"
Oct 30 20:32:04.207642 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.207585    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main"
Oct 30 20:32:04.207642 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.207593    5958 cpu_manager.go:414] "containerMap: removing container" podUID="a984e6de-54fc-4ec0-83b0-ab295078699b" containerName="payload-processing"
Oct 30 20:32:04.207642 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.207598    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a984e6de-54fc-4ec0-83b0-ab295078699b" containerName="payload-processing"
Oct 30 20:32:04.207642 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.207604    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3b96a18b-6691-4463-88dd-933e4057f7b4" containerName="revoke-keys"
Oct 30 20:32:04.207642 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.207609    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b96a18b-6691-4463-88dd-933e4057f7b4" containerName="revoke-keys"
Oct 30 20:32:04.208029 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.207668    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c707594b-f9af-4025-8614-43ce0feaa1e9" containerName="istio-proxy"
Oct 30 20:32:04.208029 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.207675    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="504cc4c3-8577-43e6-8feb-5263e0d7d310" containerName="main"
Oct 30 20:32:04.208029 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.207683    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="a984e6de-54fc-4ec0-83b0-ab295078699b" containerName="payload-processing"
Oct 30 20:32:04.208029 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.207691    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b96a18b-6691-4463-88dd-933e4057f7b4" containerName="revoke-keys"
Oct 30 20:32:04.211968 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.211950    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nm9rf"
Oct 30 20:32:04.274411 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.274385    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:32:04.274542 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.274510    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:32:04.290128 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.290094    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5370ed05-f050-4c48-a623-17239dd5408d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-nm9rf\" (UID: \"5370ed05-f050-4c48-a623-17239dd5408d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nm9rf"
Oct 30 20:32:04.290266 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.290155    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5370ed05-f050-4c48-a623-17239dd5408d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-nm9rf\" (UID: \"5370ed05-f050-4c48-a623-17239dd5408d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nm9rf"
Oct 30 20:32:04.290266 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.290186    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l5kk\" (UniqueName: \"kubernetes.io/projected/5370ed05-f050-4c48-a623-17239dd5408d-kube-api-access-4l5kk\") pod \"maas-api-revoke-sub-85237e45-nm9rf\" (UID: \"5370ed05-f050-4c48-a623-17239dd5408d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nm9rf"
Oct 30 20:32:04.324466 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.324430    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:32:04.324586 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.324512    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:32:04.325965 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.325944    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nm9rf"]
Oct 30 20:32:04.391125 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.391102    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5370ed05-f050-4c48-a623-17239dd5408d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-nm9rf\" (UID: \"5370ed05-f050-4c48-a623-17239dd5408d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nm9rf"
Oct 30 20:32:04.391240 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.391144    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5370ed05-f050-4c48-a623-17239dd5408d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-nm9rf\" (UID: \"5370ed05-f050-4c48-a623-17239dd5408d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nm9rf"
Oct 30 20:32:04.391240 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.391185    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l5kk\" (UniqueName: \"kubernetes.io/projected/5370ed05-f050-4c48-a623-17239dd5408d-kube-api-access-4l5kk\") pod \"maas-api-revoke-sub-85237e45-nm9rf\" (UID: \"5370ed05-f050-4c48-a623-17239dd5408d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nm9rf"
Oct 30 20:32:04.391770 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.391752    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5370ed05-f050-4c48-a623-17239dd5408d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-nm9rf\" (UID: \"5370ed05-f050-4c48-a623-17239dd5408d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nm9rf"
Oct 30 20:32:04.393696 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.393676    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5370ed05-f050-4c48-a623-17239dd5408d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-nm9rf\" (UID: \"5370ed05-f050-4c48-a623-17239dd5408d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nm9rf"
Oct 30 20:32:04.452842 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.452823    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l5kk\" (UniqueName: \"kubernetes.io/projected/5370ed05-f050-4c48-a623-17239dd5408d-kube-api-access-4l5kk\") pod \"maas-api-revoke-sub-85237e45-nm9rf\" (UID: \"5370ed05-f050-4c48-a623-17239dd5408d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nm9rf"
Oct 30 20:32:04.525058 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.525008    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nm9rf"
Oct 30 20:32:04.896877 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.896853    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nm9rf"]
Oct 30 20:32:04.899439 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:32:04.899411    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5370ed05_f050_4c48_a623_17239dd5408d.slice/crio-61a87bec61bde82c992adc77738568a0ccd2fe237fef43874dee99053c116abc WatchSource:0}: Error finding container 61a87bec61bde82c992adc77738568a0ccd2fe237fef43874dee99053c116abc: Status 404 returned error can't find the container with id 61a87bec61bde82c992adc77738568a0ccd2fe237fef43874dee99053c116abc
Oct 30 20:32:04.943543 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:04.943486    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nm9rf" event={"ID":"5370ed05-f050-4c48-a623-17239dd5408d","Type":"ContainerStarted","Data":"61a87bec61bde82c992adc77738568a0ccd2fe237fef43874dee99053c116abc"}
Oct 30 20:32:05.950692 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:05.950655    5958 generic.go:356] "Generic (PLEG): container finished" podID="5370ed05-f050-4c48-a623-17239dd5408d" containerID="beec1fa9102c26c7900a37f93856dee16d256b8ee3835f653df7dee4d785b69c" exitCode=0
Oct 30 20:32:05.951035 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:05.950715    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nm9rf" event={"ID":"5370ed05-f050-4c48-a623-17239dd5408d","Type":"ContainerDied","Data":"beec1fa9102c26c7900a37f93856dee16d256b8ee3835f653df7dee4d785b69c"}
Oct 30 20:32:06.603927 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:32:06.603890    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:32:07.086814 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:07.086792    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nm9rf"
Oct 30 20:32:07.111029 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:07.111001    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5370ed05-f050-4c48-a623-17239dd5408d-kube-api-access-4l5kk\" (UniqueName: \"kubernetes.io/projected/5370ed05-f050-4c48-a623-17239dd5408d-kube-api-access-4l5kk\") pod \"5370ed05-f050-4c48-a623-17239dd5408d\" (UID: \"5370ed05-f050-4c48-a623-17239dd5408d\") "
Oct 30 20:32:07.111136 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:07.111068    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5370ed05-f050-4c48-a623-17239dd5408d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5370ed05-f050-4c48-a623-17239dd5408d-maas-api-openshift-ca\") pod \"5370ed05-f050-4c48-a623-17239dd5408d\" (UID: \"5370ed05-f050-4c48-a623-17239dd5408d\") "
Oct 30 20:32:07.111182 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:07.111141    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5370ed05-f050-4c48-a623-17239dd5408d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5370ed05-f050-4c48-a623-17239dd5408d-maas-api-xks-ca\") pod \"5370ed05-f050-4c48-a623-17239dd5408d\" (UID: \"5370ed05-f050-4c48-a623-17239dd5408d\") "
Oct 30 20:32:07.111408 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:07.111384    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5370ed05-f050-4c48-a623-17239dd5408d-maas-api-openshift-ca" pod "5370ed05-f050-4c48-a623-17239dd5408d" (UID: "5370ed05-f050-4c48-a623-17239dd5408d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:32:07.111559 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:07.111541    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5370ed05-f050-4c48-a623-17239dd5408d-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:32:07.113161 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:07.113141    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5370ed05-f050-4c48-a623-17239dd5408d-kube-api-access-4l5kk" pod "5370ed05-f050-4c48-a623-17239dd5408d" (UID: "5370ed05-f050-4c48-a623-17239dd5408d"). InnerVolumeSpecName "kube-api-access-4l5kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:32:07.113253 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:07.113185    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5370ed05-f050-4c48-a623-17239dd5408d-maas-api-xks-ca" pod "5370ed05-f050-4c48-a623-17239dd5408d" (UID: "5370ed05-f050-4c48-a623-17239dd5408d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:32:07.212422 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:07.212364    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5370ed05-f050-4c48-a623-17239dd5408d-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:32:07.212422 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:07.212392    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l5kk\" (UniqueName: \"kubernetes.io/projected/5370ed05-f050-4c48-a623-17239dd5408d-kube-api-access-4l5kk\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:32:07.962236 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:07.962211    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nm9rf"
Oct 30 20:32:07.962391 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:07.962206    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nm9rf" event={"ID":"5370ed05-f050-4c48-a623-17239dd5408d","Type":"ContainerDied","Data":"61a87bec61bde82c992adc77738568a0ccd2fe237fef43874dee99053c116abc"}
Oct 30 20:32:07.962391 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:07.962340    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61a87bec61bde82c992adc77738568a0ccd2fe237fef43874dee99053c116abc"
Oct 30 20:32:16.987477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:16.987424    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"]
Oct 30 20:32:16.988317 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:16.988286    5958 cpu_manager.go:414] "containerMap: removing container" podUID="5370ed05-f050-4c48-a623-17239dd5408d" containerName="revoke-keys"
Oct 30 20:32:16.988317 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:16.988312    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5370ed05-f050-4c48-a623-17239dd5408d" containerName="revoke-keys"
Oct 30 20:32:16.988495 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:16.988411    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="5370ed05-f050-4c48-a623-17239dd5408d" containerName="revoke-keys"
Oct 30 20:32:16.991990 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:16.991972    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.096436 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.096399    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"]
Oct 30 20:32:17.111411 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.111343    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1d3194-b173-4350-95c7-3bba8f068267-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.111773 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.111749    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1d3194-b173-4350-95c7-3bba8f068267-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.111911 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.111892    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f1d3194-b173-4350-95c7-3bba8f068267-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.112000 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.111932    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f1d3194-b173-4350-95c7-3bba8f068267-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.112058 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.112031    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f1d3194-b173-4350-95c7-3bba8f068267-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.112140 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.112125    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f1d3194-b173-4350-95c7-3bba8f068267-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.112201 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.112179    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f1d3194-b173-4350-95c7-3bba8f068267-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.112282 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.112266    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmql7\" (UniqueName: \"kubernetes.io/projected/5f1d3194-b173-4350-95c7-3bba8f068267-kube-api-access-cmql7\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.112332 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.112295    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f1d3194-b173-4350-95c7-3bba8f068267-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.213370 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.213337    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f1d3194-b173-4350-95c7-3bba8f068267-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.213370 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.213373    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f1d3194-b173-4350-95c7-3bba8f068267-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.213609 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.213405    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f1d3194-b173-4350-95c7-3bba8f068267-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.213609 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.213430    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f1d3194-b173-4350-95c7-3bba8f068267-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.213609 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.213501    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f1d3194-b173-4350-95c7-3bba8f068267-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.213609 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.213550    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmql7\" (UniqueName: \"kubernetes.io/projected/5f1d3194-b173-4350-95c7-3bba8f068267-kube-api-access-cmql7\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.213609 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.213570    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f1d3194-b173-4350-95c7-3bba8f068267-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.213609 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.213600    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1d3194-b173-4350-95c7-3bba8f068267-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.213925 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.213643    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1d3194-b173-4350-95c7-3bba8f068267-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.213925 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.213858    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f1d3194-b173-4350-95c7-3bba8f068267-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.214058 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.214032    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f1d3194-b173-4350-95c7-3bba8f068267-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.214149 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.214125    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1d3194-b173-4350-95c7-3bba8f068267-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.214225 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.214142    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1d3194-b173-4350-95c7-3bba8f068267-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.214225 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.214209    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f1d3194-b173-4350-95c7-3bba8f068267-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.215967 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.215945    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f1d3194-b173-4350-95c7-3bba8f068267-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.216236 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.216217    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f1d3194-b173-4350-95c7-3bba8f068267-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.275511 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.275420    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmql7\" (UniqueName: \"kubernetes.io/projected/5f1d3194-b173-4350-95c7-3bba8f068267-kube-api-access-cmql7\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.275676 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.275634    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f1d3194-b173-4350-95c7-3bba8f068267-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ds7rd\" (UID: \"5f1d3194-b173-4350-95c7-3bba8f068267\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.304833 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.304805    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:17.736505 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.736474    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"]
Oct 30 20:32:17.737889 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:32:17.737844    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f1d3194_b173_4350_95c7_3bba8f068267.slice/crio-62d4fc4afc6596c5e66058c533967dfd9297e5c04791603040b44537461ab417 WatchSource:0}: Error finding container 62d4fc4afc6596c5e66058c533967dfd9297e5c04791603040b44537461ab417: Status 404 returned error can't find the container with id 62d4fc4afc6596c5e66058c533967dfd9297e5c04791603040b44537461ab417
Oct 30 20:32:17.740275 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.740231    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:32:17.740389 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.740316    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:32:17.740389 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:17.740352    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:32:18.012579 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:18.012482    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd" event={"ID":"5f1d3194-b173-4350-95c7-3bba8f068267","Type":"ContainerStarted","Data":"abc607b016129acc57b1f48ae3cdb316e3b1462d40b254655c472ab1fdc5e84a"}
Oct 30 20:32:18.012579 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:18.012555    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd" event={"ID":"5f1d3194-b173-4350-95c7-3bba8f068267","Type":"ContainerStarted","Data":"62d4fc4afc6596c5e66058c533967dfd9297e5c04791603040b44537461ab417"}
Oct 30 20:32:18.134771 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:18.134722    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd" podStartSLOduration=2.134708372 podStartE2EDuration="2.134708372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:18.13354498 +0000 UTC m=+2976.865760370" watchObservedRunningTime="2026-10-30 20:32:18.134708372 +0000 UTC m=+2976.866923760"
Oct 30 20:32:18.305561 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:18.305490    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:18.307214 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:18.307186    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 20:32:18.307362 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:18.307240    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd" podUID="5f1d3194-b173-4350-95c7-3bba8f068267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 20:32:19.305551 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:19.305517    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 20:32:19.305921 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:19.305577    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd" podUID="5f1d3194-b173-4350-95c7-3bba8f068267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 20:32:20.062581 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:20.062498    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4be744cb-vh4h7"]
Oct 30 20:32:20.112303 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:20.112276    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4be744cb-vh4h7"]
Oct 30 20:32:20.305600 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:20.305569    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 20:32:20.305991 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:20.305619    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd" podUID="5f1d3194-b173-4350-95c7-3bba8f068267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 20:32:21.305955 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:21.305909    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 20:32:21.306339 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:21.305973    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd" podUID="5f1d3194-b173-4350-95c7-3bba8f068267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 20:32:21.398660 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:21.398624    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b26180f-b498-4cc0-97e0-4b906a22f2f6" path="/var/lib/kubelet/pods/9b26180f-b498-4cc0-97e0-4b906a22f2f6/volumes"
Oct 30 20:32:22.305947 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:22.305911    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 20:32:22.306107 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:22.305972    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd" podUID="5f1d3194-b173-4350-95c7-3bba8f068267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 20:32:23.305894 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:23.305858    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 20:32:23.306111 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:23.305918    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd" podUID="5f1d3194-b173-4350-95c7-3bba8f068267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 20:32:24.306237 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:24.306200    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 20:32:24.306671 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:24.306262    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd" podUID="5f1d3194-b173-4350-95c7-3bba8f068267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 20:32:25.745230 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:25.745194    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:25.745770 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:25.745564    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:25.746562 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:25.746537    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ds7rd"
Oct 30 20:32:36.605376 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:32:36.605339    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:32:41.908391 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:41.908363    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 20:32:41.921690 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:32:41.921665    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 20:33:03.986991 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:03.986951    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vdpxl"]
Oct 30 20:33:03.991659 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:03.991640    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vdpxl"
Oct 30 20:33:04.048369 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:04.048333    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vdpxl"]
Oct 30 20:33:04.086233 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:04.086193    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/837b36db-1b4d-46b5-85f0-2132c326f5e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vdpxl\" (UID: \"837b36db-1b4d-46b5-85f0-2132c326f5e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vdpxl"
Oct 30 20:33:04.086233 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:04.086237    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/837b36db-1b4d-46b5-85f0-2132c326f5e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vdpxl\" (UID: \"837b36db-1b4d-46b5-85f0-2132c326f5e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vdpxl"
Oct 30 20:33:04.086523 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:04.086440    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkmht\" (UniqueName: \"kubernetes.io/projected/837b36db-1b4d-46b5-85f0-2132c326f5e8-kube-api-access-mkmht\") pod \"maas-api-revoke-sub-c1e271a9-vdpxl\" (UID: \"837b36db-1b4d-46b5-85f0-2132c326f5e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vdpxl"
Oct 30 20:33:04.096083 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:04.096058    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:33:04.096202 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:04.096054    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:33:04.096202 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:04.096097    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:33:04.096202 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:04.096131    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:33:04.187603 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:04.187559    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/837b36db-1b4d-46b5-85f0-2132c326f5e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vdpxl\" (UID: \"837b36db-1b4d-46b5-85f0-2132c326f5e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vdpxl"
Oct 30 20:33:04.187603 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:04.187606    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/837b36db-1b4d-46b5-85f0-2132c326f5e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vdpxl\" (UID: \"837b36db-1b4d-46b5-85f0-2132c326f5e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vdpxl"
Oct 30 20:33:04.187858 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:04.187688    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkmht\" (UniqueName: \"kubernetes.io/projected/837b36db-1b4d-46b5-85f0-2132c326f5e8-kube-api-access-mkmht\") pod \"maas-api-revoke-sub-c1e271a9-vdpxl\" (UID: \"837b36db-1b4d-46b5-85f0-2132c326f5e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vdpxl"
Oct 30 20:33:04.188216 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:04.188186    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/837b36db-1b4d-46b5-85f0-2132c326f5e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vdpxl\" (UID: \"837b36db-1b4d-46b5-85f0-2132c326f5e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vdpxl"
Oct 30 20:33:04.190233 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:04.190210    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/837b36db-1b4d-46b5-85f0-2132c326f5e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vdpxl\" (UID: \"837b36db-1b4d-46b5-85f0-2132c326f5e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vdpxl"
Oct 30 20:33:04.249307 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:04.249232    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkmht\" (UniqueName: \"kubernetes.io/projected/837b36db-1b4d-46b5-85f0-2132c326f5e8-kube-api-access-mkmht\") pod \"maas-api-revoke-sub-c1e271a9-vdpxl\" (UID: \"837b36db-1b4d-46b5-85f0-2132c326f5e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vdpxl"
Oct 30 20:33:04.304139 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:04.304105    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vdpxl"
Oct 30 20:33:04.677923 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:04.677899    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vdpxl"]
Oct 30 20:33:04.680737 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:33:04.680709    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837b36db_1b4d_46b5_85f0_2132c326f5e8.slice/crio-c630a16c275a13c1066bd257f9ca721f66c4c00d66f9d16b66e62218c591fec2 WatchSource:0}: Error finding container c630a16c275a13c1066bd257f9ca721f66c4c00d66f9d16b66e62218c591fec2: Status 404 returned error can't find the container with id c630a16c275a13c1066bd257f9ca721f66c4c00d66f9d16b66e62218c591fec2
Oct 30 20:33:05.270255 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:05.270153    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vdpxl" event={"ID":"837b36db-1b4d-46b5-85f0-2132c326f5e8","Type":"ContainerStarted","Data":"7efac8c5f3a17d5f69005be6778c918156d710494c76a05ea239e6f345eebfcc"}
Oct 30 20:33:05.270255 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:05.270214    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vdpxl" event={"ID":"837b36db-1b4d-46b5-85f0-2132c326f5e8","Type":"ContainerStarted","Data":"c630a16c275a13c1066bd257f9ca721f66c4c00d66f9d16b66e62218c591fec2"}
Oct 30 20:33:05.387417 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:05.387345    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vdpxl" podStartSLOduration=2.387332269 podStartE2EDuration="2.387332269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:05.385999235 +0000 UTC m=+3024.118214624" watchObservedRunningTime="2026-10-30 20:33:05.387332269 +0000 UTC m=+3024.119547652"
Oct 30 20:33:06.277112 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:06.277072    5958 generic.go:356] "Generic (PLEG): container finished" podID="837b36db-1b4d-46b5-85f0-2132c326f5e8" containerID="7efac8c5f3a17d5f69005be6778c918156d710494c76a05ea239e6f345eebfcc" exitCode=0
Oct 30 20:33:06.277561 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:06.277150    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vdpxl" event={"ID":"837b36db-1b4d-46b5-85f0-2132c326f5e8","Type":"ContainerDied","Data":"7efac8c5f3a17d5f69005be6778c918156d710494c76a05ea239e6f345eebfcc"}
Oct 30 20:33:06.605237 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:33:06.605201    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:33:07.413710 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:07.413684    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vdpxl"
Oct 30 20:33:07.520065 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:07.520039    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/837b36db-1b4d-46b5-85f0-2132c326f5e8-kube-api-access-mkmht\" (UniqueName: \"kubernetes.io/projected/837b36db-1b4d-46b5-85f0-2132c326f5e8-kube-api-access-mkmht\") pod \"837b36db-1b4d-46b5-85f0-2132c326f5e8\" (UID: \"837b36db-1b4d-46b5-85f0-2132c326f5e8\") "
Oct 30 20:33:07.520188 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:07.520088    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/837b36db-1b4d-46b5-85f0-2132c326f5e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/837b36db-1b4d-46b5-85f0-2132c326f5e8-maas-api-openshift-ca\") pod \"837b36db-1b4d-46b5-85f0-2132c326f5e8\" (UID: \"837b36db-1b4d-46b5-85f0-2132c326f5e8\") "
Oct 30 20:33:07.520188 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:07.520138    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/837b36db-1b4d-46b5-85f0-2132c326f5e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/837b36db-1b4d-46b5-85f0-2132c326f5e8-maas-api-xks-ca\") pod \"837b36db-1b4d-46b5-85f0-2132c326f5e8\" (UID: \"837b36db-1b4d-46b5-85f0-2132c326f5e8\") "
Oct 30 20:33:07.520503 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:07.520476    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837b36db-1b4d-46b5-85f0-2132c326f5e8-maas-api-openshift-ca" pod "837b36db-1b4d-46b5-85f0-2132c326f5e8" (UID: "837b36db-1b4d-46b5-85f0-2132c326f5e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:33:07.520605 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:07.520566    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/837b36db-1b4d-46b5-85f0-2132c326f5e8-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:33:07.522253 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:07.522232    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837b36db-1b4d-46b5-85f0-2132c326f5e8-kube-api-access-mkmht" pod "837b36db-1b4d-46b5-85f0-2132c326f5e8" (UID: "837b36db-1b4d-46b5-85f0-2132c326f5e8"). InnerVolumeSpecName "kube-api-access-mkmht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:33:07.522357 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:07.522277    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/837b36db-1b4d-46b5-85f0-2132c326f5e8-maas-api-xks-ca" pod "837b36db-1b4d-46b5-85f0-2132c326f5e8" (UID: "837b36db-1b4d-46b5-85f0-2132c326f5e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:33:07.621282 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:07.621257    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/837b36db-1b4d-46b5-85f0-2132c326f5e8-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:33:07.621400 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:07.621283    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkmht\" (UniqueName: \"kubernetes.io/projected/837b36db-1b4d-46b5-85f0-2132c326f5e8-kube-api-access-mkmht\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:33:08.289287 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:08.289258    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vdpxl"
Oct 30 20:33:08.289469 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:08.289282    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vdpxl" event={"ID":"837b36db-1b4d-46b5-85f0-2132c326f5e8","Type":"ContainerDied","Data":"c630a16c275a13c1066bd257f9ca721f66c4c00d66f9d16b66e62218c591fec2"}
Oct 30 20:33:08.289469 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:08.289347    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c630a16c275a13c1066bd257f9ca721f66c4c00d66f9d16b66e62218c591fec2"
Oct 30 20:33:17.595787 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.595752    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"]
Oct 30 20:33:17.596397 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.596189    5958 cpu_manager.go:414] "containerMap: removing container" podUID="837b36db-1b4d-46b5-85f0-2132c326f5e8" containerName="revoke-keys"
Oct 30 20:33:17.596397 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.596200    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="837b36db-1b4d-46b5-85f0-2132c326f5e8" containerName="revoke-keys"
Oct 30 20:33:17.596397 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.596288    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="837b36db-1b4d-46b5-85f0-2132c326f5e8" containerName="revoke-keys"
Oct 30 20:33:17.599419 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.599398    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.700536 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.700496    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/496d86d7-0610-4b19-803d-44ef7ce1fe6b-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.700713 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.700566    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/496d86d7-0610-4b19-803d-44ef7ce1fe6b-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.700713 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.700613    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/496d86d7-0610-4b19-803d-44ef7ce1fe6b-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.700713 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.700637    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gnqr\" (UniqueName: \"kubernetes.io/projected/496d86d7-0610-4b19-803d-44ef7ce1fe6b-kube-api-access-5gnqr\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.700713 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.700672    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/496d86d7-0610-4b19-803d-44ef7ce1fe6b-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.700713 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.700697    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/496d86d7-0610-4b19-803d-44ef7ce1fe6b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.700713 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.700712    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/496d86d7-0610-4b19-803d-44ef7ce1fe6b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.701047 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.700770    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/496d86d7-0610-4b19-803d-44ef7ce1fe6b-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.701047 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.700798    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/496d86d7-0610-4b19-803d-44ef7ce1fe6b-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.704799 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.704769    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"]
Oct 30 20:33:17.801709 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.801675    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/496d86d7-0610-4b19-803d-44ef7ce1fe6b-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.801876 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.801717    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/496d86d7-0610-4b19-803d-44ef7ce1fe6b-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.801876 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.801793    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/496d86d7-0610-4b19-803d-44ef7ce1fe6b-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.801979 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.801939    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/496d86d7-0610-4b19-803d-44ef7ce1fe6b-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.802036 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.801993    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/496d86d7-0610-4b19-803d-44ef7ce1fe6b-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.802036 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.802025    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gnqr\" (UniqueName: \"kubernetes.io/projected/496d86d7-0610-4b19-803d-44ef7ce1fe6b-kube-api-access-5gnqr\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.802154 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.802064    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/496d86d7-0610-4b19-803d-44ef7ce1fe6b-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.802154 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.802090    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/496d86d7-0610-4b19-803d-44ef7ce1fe6b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.802154 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.802111    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/496d86d7-0610-4b19-803d-44ef7ce1fe6b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.802316 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.802224    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/496d86d7-0610-4b19-803d-44ef7ce1fe6b-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.802316 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.802273    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/496d86d7-0610-4b19-803d-44ef7ce1fe6b-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.802444 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.802367    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/496d86d7-0610-4b19-803d-44ef7ce1fe6b-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.802604 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.802555    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/496d86d7-0610-4b19-803d-44ef7ce1fe6b-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.802811 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.802782    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/496d86d7-0610-4b19-803d-44ef7ce1fe6b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.804290 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.804266    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/496d86d7-0610-4b19-803d-44ef7ce1fe6b-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.804864 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.804841    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/496d86d7-0610-4b19-803d-44ef7ce1fe6b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.872893 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.872819    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/496d86d7-0610-4b19-803d-44ef7ce1fe6b-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.873060 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.873038    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gnqr\" (UniqueName: \"kubernetes.io/projected/496d86d7-0610-4b19-803d-44ef7ce1fe6b-kube-api-access-5gnqr\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wgtq6\" (UID: \"496d86d7-0610-4b19-803d-44ef7ce1fe6b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:17.911902 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:17.911874    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:18.364582 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:18.364549    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"]
Oct 30 20:33:18.365983 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:33:18.365952    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496d86d7_0610_4b19_803d_44ef7ce1fe6b.slice/crio-a9157df254b3f282b970a3e8cfe439bd0d532dffeeb960fab0b66fd179467065 WatchSource:0}: Error finding container a9157df254b3f282b970a3e8cfe439bd0d532dffeeb960fab0b66fd179467065: Status 404 returned error can't find the container with id a9157df254b3f282b970a3e8cfe439bd0d532dffeeb960fab0b66fd179467065
Oct 30 20:33:18.368560 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:18.368524    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:33:18.368692 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:18.368591    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:33:18.368692 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:18.368620    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:33:19.366317 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:19.366271    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6" event={"ID":"496d86d7-0610-4b19-803d-44ef7ce1fe6b","Type":"ContainerStarted","Data":"2753d7e9f60e5e74c413bfdc7d6c95410a9ca78ae280ea953dfb6465f1ba46ec"}
Oct 30 20:33:19.366758 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:19.366342    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6" event={"ID":"496d86d7-0610-4b19-803d-44ef7ce1fe6b","Type":"ContainerStarted","Data":"a9157df254b3f282b970a3e8cfe439bd0d532dffeeb960fab0b66fd179467065"}
Oct 30 20:33:19.912661 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:19.912625    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:19.914024 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:19.913994    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 20:33:19.914167 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:19.914061    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6" podUID="496d86d7-0610-4b19-803d-44ef7ce1fe6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 20:33:20.913283 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:20.913251    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 20:33:20.913679 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:20.913302    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6" podUID="496d86d7-0610-4b19-803d-44ef7ce1fe6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 20:33:21.913113 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:21.913070    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 20:33:21.913288 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:21.913136    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6" podUID="496d86d7-0610-4b19-803d-44ef7ce1fe6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 20:33:22.912239 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:22.912209    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 20:33:22.912678 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:22.912259    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6" podUID="496d86d7-0610-4b19-803d-44ef7ce1fe6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 20:33:23.913087 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:23.913048    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 20:33:23.913525 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:23.913117    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6" podUID="496d86d7-0610-4b19-803d-44ef7ce1fe6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 20:33:24.912545 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:24.912507    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 20:33:24.912773 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:24.912570    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6" podUID="496d86d7-0610-4b19-803d-44ef7ce1fe6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 20:33:26.913803 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:26.913708    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:33:26.913803 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:26.913768    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6" podUID="496d86d7-0610-4b19-803d-44ef7ce1fe6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:33:27.955615 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:27.955571    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:33:27.956071 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:27.955651    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6" podUID="496d86d7-0610-4b19-803d-44ef7ce1fe6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:33:28.575537 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:28.575431    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:28.576044 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:28.576018    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:28.576726 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:28.576706    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6"
Oct 30 20:33:28.747167 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:28.747122    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wgtq6" podStartSLOduration=11.747111723 podStartE2EDuration="11.747111723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:19.560757487 +0000 UTC m=+3038.292972889" watchObservedRunningTime="2026-10-30 20:33:28.747111723 +0000 UTC m=+3047.479327111"
Oct 30 20:33:36.605518 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:33:36.605482    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:33:38.702089 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:38.702060    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4zq44"]
Oct 30 20:33:38.705946 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:38.705928    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4zq44"
Oct 30 20:33:38.758890 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:38.758862    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:33:38.758890 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:38.758862    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:33:38.809235 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:38.809208    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:33:38.809370 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:38.809238    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:33:38.810859 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:38.810838    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4zq44"]
Oct 30 20:33:38.817863 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:38.817833    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2d7d1c-da0c-449d-b351-b6668a7d4e11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4zq44\" (UID: \"2b2d7d1c-da0c-449d-b351-b6668a7d4e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4zq44"
Oct 30 20:33:38.817979 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:38.817964    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2d7d1c-da0c-449d-b351-b6668a7d4e11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4zq44\" (UID: \"2b2d7d1c-da0c-449d-b351-b6668a7d4e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4zq44"
Oct 30 20:33:38.818051 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:38.818030    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpd68\" (UniqueName: \"kubernetes.io/projected/2b2d7d1c-da0c-449d-b351-b6668a7d4e11-kube-api-access-gpd68\") pod \"maas-api-revoke-sub-2e1f5b5f-4zq44\" (UID: \"2b2d7d1c-da0c-449d-b351-b6668a7d4e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4zq44"
Oct 30 20:33:38.919489 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:38.919438    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpd68\" (UniqueName: \"kubernetes.io/projected/2b2d7d1c-da0c-449d-b351-b6668a7d4e11-kube-api-access-gpd68\") pod \"maas-api-revoke-sub-2e1f5b5f-4zq44\" (UID: \"2b2d7d1c-da0c-449d-b351-b6668a7d4e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4zq44"
Oct 30 20:33:38.919611 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:38.919569    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2d7d1c-da0c-449d-b351-b6668a7d4e11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4zq44\" (UID: \"2b2d7d1c-da0c-449d-b351-b6668a7d4e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4zq44"
Oct 30 20:33:38.919717 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:38.919660    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2d7d1c-da0c-449d-b351-b6668a7d4e11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4zq44\" (UID: \"2b2d7d1c-da0c-449d-b351-b6668a7d4e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4zq44"
Oct 30 20:33:38.920198 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:38.920176    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2d7d1c-da0c-449d-b351-b6668a7d4e11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4zq44\" (UID: \"2b2d7d1c-da0c-449d-b351-b6668a7d4e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4zq44"
Oct 30 20:33:38.922039 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:38.922017    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2d7d1c-da0c-449d-b351-b6668a7d4e11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4zq44\" (UID: \"2b2d7d1c-da0c-449d-b351-b6668a7d4e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4zq44"
Oct 30 20:33:38.981137 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:38.981085    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpd68\" (UniqueName: \"kubernetes.io/projected/2b2d7d1c-da0c-449d-b351-b6668a7d4e11-kube-api-access-gpd68\") pod \"maas-api-revoke-sub-2e1f5b5f-4zq44\" (UID: \"2b2d7d1c-da0c-449d-b351-b6668a7d4e11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4zq44"
Oct 30 20:33:39.019729 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:39.019708    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4zq44"
Oct 30 20:33:39.397849 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:39.397818    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4zq44"]
Oct 30 20:33:39.398420 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:33:39.398391    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2d7d1c_da0c_449d_b351_b6668a7d4e11.slice/crio-ac154ee954afc0e3484f5adfa1c3405aaf4c5f8237041b06db25e4fd29a83018 WatchSource:0}: Error finding container ac154ee954afc0e3484f5adfa1c3405aaf4c5f8237041b06db25e4fd29a83018: Status 404 returned error can't find the container with id ac154ee954afc0e3484f5adfa1c3405aaf4c5f8237041b06db25e4fd29a83018
Oct 30 20:33:39.478311 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:39.478286    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4zq44" event={"ID":"2b2d7d1c-da0c-449d-b351-b6668a7d4e11","Type":"ContainerStarted","Data":"ac154ee954afc0e3484f5adfa1c3405aaf4c5f8237041b06db25e4fd29a83018"}
Oct 30 20:33:40.484673 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:40.484592    5958 generic.go:356] "Generic (PLEG): container finished" podID="2b2d7d1c-da0c-449d-b351-b6668a7d4e11" containerID="164c30d28eafffe277f3372f6403de140df823929e8455648e090af1dd915494" exitCode=0
Oct 30 20:33:40.485018 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:40.484663    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4zq44" event={"ID":"2b2d7d1c-da0c-449d-b351-b6668a7d4e11","Type":"ContainerDied","Data":"164c30d28eafffe277f3372f6403de140df823929e8455648e090af1dd915494"}
Oct 30 20:33:41.622687 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:41.622659    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4zq44"
Oct 30 20:33:41.745716 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:41.745644    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b2d7d1c-da0c-449d-b351-b6668a7d4e11-kube-api-access-gpd68\" (UniqueName: \"kubernetes.io/projected/2b2d7d1c-da0c-449d-b351-b6668a7d4e11-kube-api-access-gpd68\") pod \"2b2d7d1c-da0c-449d-b351-b6668a7d4e11\" (UID: \"2b2d7d1c-da0c-449d-b351-b6668a7d4e11\") "
Oct 30 20:33:41.745858 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:41.745763    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b2d7d1c-da0c-449d-b351-b6668a7d4e11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2d7d1c-da0c-449d-b351-b6668a7d4e11-maas-api-xks-ca\") pod \"2b2d7d1c-da0c-449d-b351-b6668a7d4e11\" (UID: \"2b2d7d1c-da0c-449d-b351-b6668a7d4e11\") "
Oct 30 20:33:41.745858 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:41.745816    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b2d7d1c-da0c-449d-b351-b6668a7d4e11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2d7d1c-da0c-449d-b351-b6668a7d4e11-maas-api-openshift-ca\") pod \"2b2d7d1c-da0c-449d-b351-b6668a7d4e11\" (UID: \"2b2d7d1c-da0c-449d-b351-b6668a7d4e11\") "
Oct 30 20:33:41.746317 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:41.746289    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b2d7d1c-da0c-449d-b351-b6668a7d4e11-maas-api-openshift-ca" pod "2b2d7d1c-da0c-449d-b351-b6668a7d4e11" (UID: "2b2d7d1c-da0c-449d-b351-b6668a7d4e11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:33:41.748091 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:41.748066    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2d7d1c-da0c-449d-b351-b6668a7d4e11-kube-api-access-gpd68" pod "2b2d7d1c-da0c-449d-b351-b6668a7d4e11" (UID: "2b2d7d1c-da0c-449d-b351-b6668a7d4e11"). InnerVolumeSpecName "kube-api-access-gpd68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:33:41.748231 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:41.748150    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b2d7d1c-da0c-449d-b351-b6668a7d4e11-maas-api-xks-ca" pod "2b2d7d1c-da0c-449d-b351-b6668a7d4e11" (UID: "2b2d7d1c-da0c-449d-b351-b6668a7d4e11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:33:41.847529 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:41.847507    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b2d7d1c-da0c-449d-b351-b6668a7d4e11-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:33:41.847529 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:41.847530    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpd68\" (UniqueName: \"kubernetes.io/projected/2b2d7d1c-da0c-449d-b351-b6668a7d4e11-kube-api-access-gpd68\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:33:41.847675 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:41.847539    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b2d7d1c-da0c-449d-b351-b6668a7d4e11-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:33:42.495261 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:42.495229    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4zq44"
Oct 30 20:33:42.495428 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:42.495253    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4zq44" event={"ID":"2b2d7d1c-da0c-449d-b351-b6668a7d4e11","Type":"ContainerDied","Data":"ac154ee954afc0e3484f5adfa1c3405aaf4c5f8237041b06db25e4fd29a83018"}
Oct 30 20:33:42.495428 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:42.495307    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac154ee954afc0e3484f5adfa1c3405aaf4c5f8237041b06db25e4fd29a83018"
Oct 30 20:33:42.785709 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:42.785642    5958 scope.go:122] "RemoveContainer" containerID="dabdea84473f354f1449c19887d45c4cdbb497c4f5e8c69e7926169b77b93961"
Oct 30 20:33:46.054952 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:46.054919    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb1759f-2265d"]
Oct 30 20:33:46.104641 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:46.104606    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb1759f-2265d"]
Oct 30 20:33:47.397019 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:47.396986    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88103bf5-97bb-4a35-8906-51d55325f657" path="/var/lib/kubelet/pods/88103bf5-97bb-4a35-8906-51d55325f657/volumes"
Oct 30 20:33:51.061196 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:51.061124    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q7g68"]
Oct 30 20:33:51.061653 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:51.061591    5958 cpu_manager.go:414] "containerMap: removing container" podUID="2b2d7d1c-da0c-449d-b351-b6668a7d4e11" containerName="revoke-keys"
Oct 30 20:33:51.061653 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:51.061615    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b2d7d1c-da0c-449d-b351-b6668a7d4e11" containerName="revoke-keys"
Oct 30 20:33:51.061737 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:51.061703    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b2d7d1c-da0c-449d-b351-b6668a7d4e11" containerName="revoke-keys"
Oct 30 20:33:51.066084 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:51.066069    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q7g68"
Oct 30 20:33:51.142387 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:51.142348    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:33:51.142387 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:51.142368    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:33:51.142387 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:51.142371    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:33:51.142607 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:51.142415    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:33:51.171373 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:51.171331    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q7g68"]
Oct 30 20:33:51.229750 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:51.229727    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsx5h\" (UniqueName: \"kubernetes.io/projected/6517cb79-ec8b-4d28-bd56-e649e71fd225-kube-api-access-zsx5h\") pod \"maas-api-revoke-sub-ec843018-q7g68\" (UID: \"6517cb79-ec8b-4d28-bd56-e649e71fd225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q7g68"
Oct 30 20:33:51.229891 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:51.229774    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6517cb79-ec8b-4d28-bd56-e649e71fd225-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-q7g68\" (UID: \"6517cb79-ec8b-4d28-bd56-e649e71fd225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q7g68"
Oct 30 20:33:51.229947 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:51.229928    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6517cb79-ec8b-4d28-bd56-e649e71fd225-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-q7g68\" (UID: \"6517cb79-ec8b-4d28-bd56-e649e71fd225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q7g68"
Oct 30 20:33:51.330593 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:51.330572    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6517cb79-ec8b-4d28-bd56-e649e71fd225-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-q7g68\" (UID: \"6517cb79-ec8b-4d28-bd56-e649e71fd225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q7g68"
Oct 30 20:33:51.330752 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:51.330611    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsx5h\" (UniqueName: \"kubernetes.io/projected/6517cb79-ec8b-4d28-bd56-e649e71fd225-kube-api-access-zsx5h\") pod \"maas-api-revoke-sub-ec843018-q7g68\" (UID: \"6517cb79-ec8b-4d28-bd56-e649e71fd225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q7g68"
Oct 30 20:33:51.330752 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:51.330673    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6517cb79-ec8b-4d28-bd56-e649e71fd225-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-q7g68\" (UID: \"6517cb79-ec8b-4d28-bd56-e649e71fd225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q7g68"
Oct 30 20:33:51.331266 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:51.331248    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6517cb79-ec8b-4d28-bd56-e649e71fd225-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-q7g68\" (UID: \"6517cb79-ec8b-4d28-bd56-e649e71fd225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q7g68"
Oct 30 20:33:51.332907 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:51.332888    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6517cb79-ec8b-4d28-bd56-e649e71fd225-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-q7g68\" (UID: \"6517cb79-ec8b-4d28-bd56-e649e71fd225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q7g68"
Oct 30 20:33:51.392082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:51.392059    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsx5h\" (UniqueName: \"kubernetes.io/projected/6517cb79-ec8b-4d28-bd56-e649e71fd225-kube-api-access-zsx5h\") pod \"maas-api-revoke-sub-ec843018-q7g68\" (UID: \"6517cb79-ec8b-4d28-bd56-e649e71fd225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q7g68"
Oct 30 20:33:51.678659 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:51.678586    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q7g68"
Oct 30 20:33:52.054510 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:52.054489    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q7g68"]
Oct 30 20:33:52.057952 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:33:52.057925    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6517cb79_ec8b_4d28_bd56_e649e71fd225.slice/crio-2adcc71f9efde2eccebe9ed8c38098ab33584667e69be45152e4b4af414af14a WatchSource:0}: Error finding container 2adcc71f9efde2eccebe9ed8c38098ab33584667e69be45152e4b4af414af14a: Status 404 returned error can't find the container with id 2adcc71f9efde2eccebe9ed8c38098ab33584667e69be45152e4b4af414af14a
Oct 30 20:33:52.553164 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:52.553119    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q7g68" event={"ID":"6517cb79-ec8b-4d28-bd56-e649e71fd225","Type":"ContainerStarted","Data":"53981e61950139432f7ee784ddd3b071a557d63095d4fba10daab5914ad2d0b3"}
Oct 30 20:33:52.553586 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:52.553193    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q7g68" event={"ID":"6517cb79-ec8b-4d28-bd56-e649e71fd225","Type":"ContainerStarted","Data":"2adcc71f9efde2eccebe9ed8c38098ab33584667e69be45152e4b4af414af14a"}
Oct 30 20:33:52.670236 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:52.670174    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q7g68" podStartSLOduration=1.670155472 podStartE2EDuration="1.670155472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:52.669546226 +0000 UTC m=+3071.401761616" watchObservedRunningTime="2026-10-30 20:33:52.670155472 +0000 UTC m=+3071.402370861"
Oct 30 20:33:53.562785 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:53.562739    5958 generic.go:356] "Generic (PLEG): container finished" podID="6517cb79-ec8b-4d28-bd56-e649e71fd225" containerID="53981e61950139432f7ee784ddd3b071a557d63095d4fba10daab5914ad2d0b3" exitCode=0
Oct 30 20:33:53.563198 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:53.562795    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q7g68" event={"ID":"6517cb79-ec8b-4d28-bd56-e649e71fd225","Type":"ContainerDied","Data":"53981e61950139432f7ee784ddd3b071a557d63095d4fba10daab5914ad2d0b3"}
Oct 30 20:33:54.700511 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:54.700486    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q7g68"
Oct 30 20:33:54.777587 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:54.777565    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6517cb79-ec8b-4d28-bd56-e649e71fd225-kube-api-access-zsx5h\" (UniqueName: \"kubernetes.io/projected/6517cb79-ec8b-4d28-bd56-e649e71fd225-kube-api-access-zsx5h\") pod \"6517cb79-ec8b-4d28-bd56-e649e71fd225\" (UID: \"6517cb79-ec8b-4d28-bd56-e649e71fd225\") "
Oct 30 20:33:54.777718 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:54.777658    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6517cb79-ec8b-4d28-bd56-e649e71fd225-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6517cb79-ec8b-4d28-bd56-e649e71fd225-maas-api-xks-ca\") pod \"6517cb79-ec8b-4d28-bd56-e649e71fd225\" (UID: \"6517cb79-ec8b-4d28-bd56-e649e71fd225\") "
Oct 30 20:33:54.777718 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:54.777678    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6517cb79-ec8b-4d28-bd56-e649e71fd225-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6517cb79-ec8b-4d28-bd56-e649e71fd225-maas-api-openshift-ca\") pod \"6517cb79-ec8b-4d28-bd56-e649e71fd225\" (UID: \"6517cb79-ec8b-4d28-bd56-e649e71fd225\") "
Oct 30 20:33:54.778082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:54.778060    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6517cb79-ec8b-4d28-bd56-e649e71fd225-maas-api-openshift-ca" pod "6517cb79-ec8b-4d28-bd56-e649e71fd225" (UID: "6517cb79-ec8b-4d28-bd56-e649e71fd225"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:33:54.779668 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:54.779648    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6517cb79-ec8b-4d28-bd56-e649e71fd225-kube-api-access-zsx5h" pod "6517cb79-ec8b-4d28-bd56-e649e71fd225" (UID: "6517cb79-ec8b-4d28-bd56-e649e71fd225"). InnerVolumeSpecName "kube-api-access-zsx5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:33:54.779766 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:54.779749    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6517cb79-ec8b-4d28-bd56-e649e71fd225-maas-api-xks-ca" pod "6517cb79-ec8b-4d28-bd56-e649e71fd225" (UID: "6517cb79-ec8b-4d28-bd56-e649e71fd225"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:33:54.878587 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:54.878525    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsx5h\" (UniqueName: \"kubernetes.io/projected/6517cb79-ec8b-4d28-bd56-e649e71fd225-kube-api-access-zsx5h\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:33:54.878587 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:54.878562    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6517cb79-ec8b-4d28-bd56-e649e71fd225-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:33:54.878587 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:54.878574    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6517cb79-ec8b-4d28-bd56-e649e71fd225-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:33:55.573729 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:55.573696    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q7g68" event={"ID":"6517cb79-ec8b-4d28-bd56-e649e71fd225","Type":"ContainerDied","Data":"2adcc71f9efde2eccebe9ed8c38098ab33584667e69be45152e4b4af414af14a"}
Oct 30 20:33:55.573890 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:55.573747    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2adcc71f9efde2eccebe9ed8c38098ab33584667e69be45152e4b4af414af14a"
Oct 30 20:33:55.573890 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:33:55.573706    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q7g68"
Oct 30 20:34:06.605105 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:34:06.605062    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:34:29.990650 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:29.990616    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s84lt"]
Oct 30 20:34:29.991063 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:29.991028    5958 cpu_manager.go:414] "containerMap: removing container" podUID="6517cb79-ec8b-4d28-bd56-e649e71fd225" containerName="revoke-keys"
Oct 30 20:34:29.991063 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:29.991039    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6517cb79-ec8b-4d28-bd56-e649e71fd225" containerName="revoke-keys"
Oct 30 20:34:29.991141 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:29.991100    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="6517cb79-ec8b-4d28-bd56-e649e71fd225" containerName="revoke-keys"
Oct 30 20:34:29.994399 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:29.994382    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s84lt"
Oct 30 20:34:30.096991 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:30.096964    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:34:30.097113 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:30.097001    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:34:30.097113 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:30.097020    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:34:30.097113 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:30.097006    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:34:30.098771 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:30.098753    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s84lt"]
Oct 30 20:34:30.141366 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:30.141341    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4427003a-268c-42e4-9dc8-6ba925c73504-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-s84lt\" (UID: \"4427003a-268c-42e4-9dc8-6ba925c73504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s84lt"
Oct 30 20:34:30.141509 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:30.141414    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4427003a-268c-42e4-9dc8-6ba925c73504-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-s84lt\" (UID: \"4427003a-268c-42e4-9dc8-6ba925c73504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s84lt"
Oct 30 20:34:30.141509 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:30.141498    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t64pc\" (UniqueName: \"kubernetes.io/projected/4427003a-268c-42e4-9dc8-6ba925c73504-kube-api-access-t64pc\") pod \"maas-api-revoke-sub-046e1343-s84lt\" (UID: \"4427003a-268c-42e4-9dc8-6ba925c73504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s84lt"
Oct 30 20:34:30.242395 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:30.242339    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4427003a-268c-42e4-9dc8-6ba925c73504-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-s84lt\" (UID: \"4427003a-268c-42e4-9dc8-6ba925c73504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s84lt"
Oct 30 20:34:30.242509 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:30.242405    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4427003a-268c-42e4-9dc8-6ba925c73504-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-s84lt\" (UID: \"4427003a-268c-42e4-9dc8-6ba925c73504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s84lt"
Oct 30 20:34:30.242509 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:30.242433    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t64pc\" (UniqueName: \"kubernetes.io/projected/4427003a-268c-42e4-9dc8-6ba925c73504-kube-api-access-t64pc\") pod \"maas-api-revoke-sub-046e1343-s84lt\" (UID: \"4427003a-268c-42e4-9dc8-6ba925c73504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s84lt"
Oct 30 20:34:30.243020 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:30.243001    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4427003a-268c-42e4-9dc8-6ba925c73504-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-s84lt\" (UID: \"4427003a-268c-42e4-9dc8-6ba925c73504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s84lt"
Oct 30 20:34:30.244927 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:30.244903    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4427003a-268c-42e4-9dc8-6ba925c73504-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-s84lt\" (UID: \"4427003a-268c-42e4-9dc8-6ba925c73504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s84lt"
Oct 30 20:34:30.303958 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:30.303935    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t64pc\" (UniqueName: \"kubernetes.io/projected/4427003a-268c-42e4-9dc8-6ba925c73504-kube-api-access-t64pc\") pod \"maas-api-revoke-sub-046e1343-s84lt\" (UID: \"4427003a-268c-42e4-9dc8-6ba925c73504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s84lt"
Oct 30 20:34:30.307265 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:30.307248    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s84lt"
Oct 30 20:34:30.680053 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:30.680026    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s84lt"]
Oct 30 20:34:30.681627 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:34:30.681591    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4427003a_268c_42e4_9dc8_6ba925c73504.slice/crio-207aaeb77ff073799090f9f5208e77f13551e91a2ca89649a619e1ba0917cf58 WatchSource:0}: Error finding container 207aaeb77ff073799090f9f5208e77f13551e91a2ca89649a619e1ba0917cf58: Status 404 returned error can't find the container with id 207aaeb77ff073799090f9f5208e77f13551e91a2ca89649a619e1ba0917cf58
Oct 30 20:34:30.803010 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:30.802984    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s84lt" event={"ID":"4427003a-268c-42e4-9dc8-6ba925c73504","Type":"ContainerStarted","Data":"207aaeb77ff073799090f9f5208e77f13551e91a2ca89649a619e1ba0917cf58"}
Oct 30 20:34:31.809270 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:31.809202    5958 generic.go:356] "Generic (PLEG): container finished" podID="4427003a-268c-42e4-9dc8-6ba925c73504" containerID="44fa94d7a832bf97f8cf5a399de7e53b4394a915788cb758d57a0e205bdc7c6c" exitCode=0
Oct 30 20:34:31.809640 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:31.809292    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s84lt" event={"ID":"4427003a-268c-42e4-9dc8-6ba925c73504","Type":"ContainerDied","Data":"44fa94d7a832bf97f8cf5a399de7e53b4394a915788cb758d57a0e205bdc7c6c"}
Oct 30 20:34:32.942007 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:32.941983    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s84lt"
Oct 30 20:34:32.964207 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:32.964176    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4427003a-268c-42e4-9dc8-6ba925c73504-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4427003a-268c-42e4-9dc8-6ba925c73504-maas-api-xks-ca\") pod \"4427003a-268c-42e4-9dc8-6ba925c73504\" (UID: \"4427003a-268c-42e4-9dc8-6ba925c73504\") "
Oct 30 20:34:32.964313 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:32.964264    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4427003a-268c-42e4-9dc8-6ba925c73504-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4427003a-268c-42e4-9dc8-6ba925c73504-maas-api-openshift-ca\") pod \"4427003a-268c-42e4-9dc8-6ba925c73504\" (UID: \"4427003a-268c-42e4-9dc8-6ba925c73504\") "
Oct 30 20:34:32.964313 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:32.964304    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4427003a-268c-42e4-9dc8-6ba925c73504-kube-api-access-t64pc\" (UniqueName: \"kubernetes.io/projected/4427003a-268c-42e4-9dc8-6ba925c73504-kube-api-access-t64pc\") pod \"4427003a-268c-42e4-9dc8-6ba925c73504\" (UID: \"4427003a-268c-42e4-9dc8-6ba925c73504\") "
Oct 30 20:34:32.964629 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:32.964605    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4427003a-268c-42e4-9dc8-6ba925c73504-maas-api-openshift-ca" pod "4427003a-268c-42e4-9dc8-6ba925c73504" (UID: "4427003a-268c-42e4-9dc8-6ba925c73504"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:34:32.966316 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:32.966286    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4427003a-268c-42e4-9dc8-6ba925c73504-maas-api-xks-ca" pod "4427003a-268c-42e4-9dc8-6ba925c73504" (UID: "4427003a-268c-42e4-9dc8-6ba925c73504"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:34:32.966405 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:32.966358    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4427003a-268c-42e4-9dc8-6ba925c73504-kube-api-access-t64pc" pod "4427003a-268c-42e4-9dc8-6ba925c73504" (UID: "4427003a-268c-42e4-9dc8-6ba925c73504"). InnerVolumeSpecName "kube-api-access-t64pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:34:33.065401 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:33.065335    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4427003a-268c-42e4-9dc8-6ba925c73504-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:34:33.065401 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:33.065370    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4427003a-268c-42e4-9dc8-6ba925c73504-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:34:33.065401 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:33.065386    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t64pc\" (UniqueName: \"kubernetes.io/projected/4427003a-268c-42e4-9dc8-6ba925c73504-kube-api-access-t64pc\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:34:33.819439 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:33.819410    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s84lt"
Oct 30 20:34:33.819608 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:33.819412    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s84lt" event={"ID":"4427003a-268c-42e4-9dc8-6ba925c73504","Type":"ContainerDied","Data":"207aaeb77ff073799090f9f5208e77f13551e91a2ca89649a619e1ba0917cf58"}
Oct 30 20:34:33.819608 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:33.819548    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="207aaeb77ff073799090f9f5208e77f13551e91a2ca89649a619e1ba0917cf58"
Oct 30 20:34:36.603865 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:34:36.603827    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:34:40.662774 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:40.662740    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rpw7c"]
Oct 30 20:34:40.663188 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:40.663174    5958 cpu_manager.go:414] "containerMap: removing container" podUID="4427003a-268c-42e4-9dc8-6ba925c73504" containerName="revoke-keys"
Oct 30 20:34:40.663229 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:40.663189    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4427003a-268c-42e4-9dc8-6ba925c73504" containerName="revoke-keys"
Oct 30 20:34:40.663311 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:40.663302    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="4427003a-268c-42e4-9dc8-6ba925c73504" containerName="revoke-keys"
Oct 30 20:34:40.673660 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:40.673625    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rpw7c"
Oct 30 20:34:40.725356 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:40.725318    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rpw7c"]
Oct 30 20:34:40.730672 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:40.730642    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad13ff5-8499-4767-bd69-b4f5d4131188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-rpw7c\" (UID: \"0ad13ff5-8499-4767-bd69-b4f5d4131188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rpw7c"
Oct 30 20:34:40.730846 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:40.730823    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kjq9\" (UniqueName: \"kubernetes.io/projected/0ad13ff5-8499-4767-bd69-b4f5d4131188-kube-api-access-7kjq9\") pod \"maas-api-revoke-sub-d3fba056-rpw7c\" (UID: \"0ad13ff5-8499-4767-bd69-b4f5d4131188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rpw7c"
Oct 30 20:34:40.730901 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:40.730866    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad13ff5-8499-4767-bd69-b4f5d4131188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-rpw7c\" (UID: \"0ad13ff5-8499-4767-bd69-b4f5d4131188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rpw7c"
Oct 30 20:34:40.771971 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:40.771947    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:34:40.772135 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:40.771984    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:34:40.772135 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:40.772009    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:34:40.776074 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:40.776053    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:34:40.831568 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:40.831542    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kjq9\" (UniqueName: \"kubernetes.io/projected/0ad13ff5-8499-4767-bd69-b4f5d4131188-kube-api-access-7kjq9\") pod \"maas-api-revoke-sub-d3fba056-rpw7c\" (UID: \"0ad13ff5-8499-4767-bd69-b4f5d4131188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rpw7c"
Oct 30 20:34:40.831672 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:40.831581    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad13ff5-8499-4767-bd69-b4f5d4131188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-rpw7c\" (UID: \"0ad13ff5-8499-4767-bd69-b4f5d4131188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rpw7c"
Oct 30 20:34:40.831672 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:40.831626    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad13ff5-8499-4767-bd69-b4f5d4131188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-rpw7c\" (UID: \"0ad13ff5-8499-4767-bd69-b4f5d4131188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rpw7c"
Oct 30 20:34:40.832259 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:40.832202    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad13ff5-8499-4767-bd69-b4f5d4131188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-rpw7c\" (UID: \"0ad13ff5-8499-4767-bd69-b4f5d4131188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rpw7c"
Oct 30 20:34:40.834064 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:40.834020    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad13ff5-8499-4767-bd69-b4f5d4131188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-rpw7c\" (UID: \"0ad13ff5-8499-4767-bd69-b4f5d4131188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rpw7c"
Oct 30 20:34:40.892991 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:40.892960    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kjq9\" (UniqueName: \"kubernetes.io/projected/0ad13ff5-8499-4767-bd69-b4f5d4131188-kube-api-access-7kjq9\") pod \"maas-api-revoke-sub-d3fba056-rpw7c\" (UID: \"0ad13ff5-8499-4767-bd69-b4f5d4131188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rpw7c"
Oct 30 20:34:40.987812 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:40.987755    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rpw7c"
Oct 30 20:34:41.361700 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:41.361652    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rpw7c"]
Oct 30 20:34:41.364151 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:34:41.364116    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad13ff5_8499_4767_bd69_b4f5d4131188.slice/crio-055d0551727a28331185993d4cdbd64bcf74a66499f893e5007f6813fd17d352 WatchSource:0}: Error finding container 055d0551727a28331185993d4cdbd64bcf74a66499f893e5007f6813fd17d352: Status 404 returned error can't find the container with id 055d0551727a28331185993d4cdbd64bcf74a66499f893e5007f6813fd17d352
Oct 30 20:34:41.863631 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:41.863600    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rpw7c" event={"ID":"0ad13ff5-8499-4767-bd69-b4f5d4131188","Type":"ContainerStarted","Data":"56dbcdd354e9c344650fa354af71fd4f573d6d48aa0c68efb3023711eebcca57"}
Oct 30 20:34:41.863918 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:41.863650    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rpw7c" event={"ID":"0ad13ff5-8499-4767-bd69-b4f5d4131188","Type":"ContainerStarted","Data":"055d0551727a28331185993d4cdbd64bcf74a66499f893e5007f6813fd17d352"}
Oct 30 20:34:41.981799 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:41.981698    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rpw7c" podStartSLOduration=1.981683871 podStartE2EDuration="1.981683871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:34:41.981026333 +0000 UTC m=+3120.713241733" watchObservedRunningTime="2026-10-30 20:34:41.981683871 +0000 UTC m=+3120.713899260"
Oct 30 20:34:42.870033 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:42.870001    5958 generic.go:356] "Generic (PLEG): container finished" podID="0ad13ff5-8499-4767-bd69-b4f5d4131188" containerID="56dbcdd354e9c344650fa354af71fd4f573d6d48aa0c68efb3023711eebcca57" exitCode=0
Oct 30 20:34:42.870426 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:42.870090    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rpw7c" event={"ID":"0ad13ff5-8499-4767-bd69-b4f5d4131188","Type":"ContainerDied","Data":"56dbcdd354e9c344650fa354af71fd4f573d6d48aa0c68efb3023711eebcca57"}
Oct 30 20:34:44.220468 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:44.220314    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rpw7c"
Oct 30 20:34:44.271394 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:44.270721    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ad13ff5-8499-4767-bd69-b4f5d4131188-kube-api-access-7kjq9\" (UniqueName: \"kubernetes.io/projected/0ad13ff5-8499-4767-bd69-b4f5d4131188-kube-api-access-7kjq9\") pod \"0ad13ff5-8499-4767-bd69-b4f5d4131188\" (UID: \"0ad13ff5-8499-4767-bd69-b4f5d4131188\") "
Oct 30 20:34:44.271394 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:44.270837    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ad13ff5-8499-4767-bd69-b4f5d4131188-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad13ff5-8499-4767-bd69-b4f5d4131188-maas-api-xks-ca\") pod \"0ad13ff5-8499-4767-bd69-b4f5d4131188\" (UID: \"0ad13ff5-8499-4767-bd69-b4f5d4131188\") "
Oct 30 20:34:44.271394 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:44.270940    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ad13ff5-8499-4767-bd69-b4f5d4131188-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad13ff5-8499-4767-bd69-b4f5d4131188-maas-api-openshift-ca\") pod \"0ad13ff5-8499-4767-bd69-b4f5d4131188\" (UID: \"0ad13ff5-8499-4767-bd69-b4f5d4131188\") "
Oct 30 20:34:44.272169 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:44.271880    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad13ff5-8499-4767-bd69-b4f5d4131188-maas-api-openshift-ca" pod "0ad13ff5-8499-4767-bd69-b4f5d4131188" (UID: "0ad13ff5-8499-4767-bd69-b4f5d4131188"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:34:44.303610 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:44.300585    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ad13ff5-8499-4767-bd69-b4f5d4131188-maas-api-xks-ca" pod "0ad13ff5-8499-4767-bd69-b4f5d4131188" (UID: "0ad13ff5-8499-4767-bd69-b4f5d4131188"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:34:44.303610 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:44.301748    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad13ff5-8499-4767-bd69-b4f5d4131188-kube-api-access-7kjq9" pod "0ad13ff5-8499-4767-bd69-b4f5d4131188" (UID: "0ad13ff5-8499-4767-bd69-b4f5d4131188"). InnerVolumeSpecName "kube-api-access-7kjq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:34:44.373480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:44.372544    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ad13ff5-8499-4767-bd69-b4f5d4131188-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:34:44.373480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:44.372581    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ad13ff5-8499-4767-bd69-b4f5d4131188-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:34:44.373480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:44.372598    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kjq9\" (UniqueName: \"kubernetes.io/projected/0ad13ff5-8499-4767-bd69-b4f5d4131188-kube-api-access-7kjq9\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:34:44.921996 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:44.921953    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rpw7c" event={"ID":"0ad13ff5-8499-4767-bd69-b4f5d4131188","Type":"ContainerDied","Data":"055d0551727a28331185993d4cdbd64bcf74a66499f893e5007f6813fd17d352"}
Oct 30 20:34:44.922293 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:44.922272    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="055d0551727a28331185993d4cdbd64bcf74a66499f893e5007f6813fd17d352"
Oct 30 20:34:44.922527 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:34:44.922510    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rpw7c"
Oct 30 20:35:05.051362 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:05.051316    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55a47471-2p599"]
Oct 30 20:35:05.100845 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:05.100817    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55a47471-2p599"]
Oct 30 20:35:05.398199 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:05.398168    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6a190a0-8198-4463-b199-dd7b7515425c" path="/var/lib/kubelet/pods/a6a190a0-8198-4463-b199-dd7b7515425c/volumes"
Oct 30 20:35:06.604611 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:35:06.604578    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:35:15.641671 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:15.641637    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-84mb7"]
Oct 30 20:35:15.642075 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:15.642062    5958 cpu_manager.go:414] "containerMap: removing container" podUID="0ad13ff5-8499-4767-bd69-b4f5d4131188" containerName="revoke-keys"
Oct 30 20:35:15.642132 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:15.642076    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ad13ff5-8499-4767-bd69-b4f5d4131188" containerName="revoke-keys"
Oct 30 20:35:15.642178 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:15.642167    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ad13ff5-8499-4767-bd69-b4f5d4131188" containerName="revoke-keys"
Oct 30 20:35:15.645499 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:15.645481    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-84mb7"
Oct 30 20:35:15.747945 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:15.747916    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:35:15.747945 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:15.747931    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:35:15.748130 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:15.747985    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:35:15.748130 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:15.748047    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:35:15.749765 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:15.749746    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-84mb7"]
Oct 30 20:35:15.799862 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:15.799837    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a46cc1e5-5fa0-46e2-9cee-fbac39f676fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-84mb7\" (UID: \"a46cc1e5-5fa0-46e2-9cee-fbac39f676fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-84mb7"
Oct 30 20:35:15.800000 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:15.799875    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6l7p\" (UniqueName: \"kubernetes.io/projected/a46cc1e5-5fa0-46e2-9cee-fbac39f676fb-kube-api-access-x6l7p\") pod \"maas-api-revoke-sub-ae594979-84mb7\" (UID: \"a46cc1e5-5fa0-46e2-9cee-fbac39f676fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-84mb7"
Oct 30 20:35:15.800042 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:15.800002    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a46cc1e5-5fa0-46e2-9cee-fbac39f676fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-84mb7\" (UID: \"a46cc1e5-5fa0-46e2-9cee-fbac39f676fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-84mb7"
Oct 30 20:35:15.900772 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:15.900717    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a46cc1e5-5fa0-46e2-9cee-fbac39f676fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-84mb7\" (UID: \"a46cc1e5-5fa0-46e2-9cee-fbac39f676fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-84mb7"
Oct 30 20:35:15.900772 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:15.900747    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6l7p\" (UniqueName: \"kubernetes.io/projected/a46cc1e5-5fa0-46e2-9cee-fbac39f676fb-kube-api-access-x6l7p\") pod \"maas-api-revoke-sub-ae594979-84mb7\" (UID: \"a46cc1e5-5fa0-46e2-9cee-fbac39f676fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-84mb7"
Oct 30 20:35:15.900907 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:15.900819    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a46cc1e5-5fa0-46e2-9cee-fbac39f676fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-84mb7\" (UID: \"a46cc1e5-5fa0-46e2-9cee-fbac39f676fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-84mb7"
Oct 30 20:35:15.901483 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:15.901442    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a46cc1e5-5fa0-46e2-9cee-fbac39f676fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-84mb7\" (UID: \"a46cc1e5-5fa0-46e2-9cee-fbac39f676fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-84mb7"
Oct 30 20:35:15.902992 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:15.902970    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a46cc1e5-5fa0-46e2-9cee-fbac39f676fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-84mb7\" (UID: \"a46cc1e5-5fa0-46e2-9cee-fbac39f676fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-84mb7"
Oct 30 20:35:15.962016 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:15.961997    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6l7p\" (UniqueName: \"kubernetes.io/projected/a46cc1e5-5fa0-46e2-9cee-fbac39f676fb-kube-api-access-x6l7p\") pod \"maas-api-revoke-sub-ae594979-84mb7\" (UID: \"a46cc1e5-5fa0-46e2-9cee-fbac39f676fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-84mb7"
Oct 30 20:35:16.258792 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:16.258740    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-84mb7"
Oct 30 20:35:16.629229 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:16.629201    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-84mb7"]
Oct 30 20:35:16.631650 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:35:16.631618    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda46cc1e5_5fa0_46e2_9cee_fbac39f676fb.slice/crio-6993b3f55bff768dd52891e1f7dfd67fee677b648122e735fff7ee76f0b211ae WatchSource:0}: Error finding container 6993b3f55bff768dd52891e1f7dfd67fee677b648122e735fff7ee76f0b211ae: Status 404 returned error can't find the container with id 6993b3f55bff768dd52891e1f7dfd67fee677b648122e735fff7ee76f0b211ae
Oct 30 20:35:17.090617 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:17.090585    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-84mb7" event={"ID":"a46cc1e5-5fa0-46e2-9cee-fbac39f676fb","Type":"ContainerStarted","Data":"3b0c764670e6fa38e21238377b9639a692d0fb6d8cdbcd64876e2b2adea24c0e"}
Oct 30 20:35:17.091031 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:17.090647    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-84mb7" event={"ID":"a46cc1e5-5fa0-46e2-9cee-fbac39f676fb","Type":"ContainerStarted","Data":"6993b3f55bff768dd52891e1f7dfd67fee677b648122e735fff7ee76f0b211ae"}
Oct 30 20:35:17.207060 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:17.206962    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-84mb7" podStartSLOduration=2.206945275 podStartE2EDuration="2.206945275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:17.206422436 +0000 UTC m=+3155.938637826" watchObservedRunningTime="2026-10-30 20:35:17.206945275 +0000 UTC m=+3155.939160666"
Oct 30 20:35:18.097141 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:18.097106    5958 generic.go:356] "Generic (PLEG): container finished" podID="a46cc1e5-5fa0-46e2-9cee-fbac39f676fb" containerID="3b0c764670e6fa38e21238377b9639a692d0fb6d8cdbcd64876e2b2adea24c0e" exitCode=0
Oct 30 20:35:18.097522 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:18.097194    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-84mb7" event={"ID":"a46cc1e5-5fa0-46e2-9cee-fbac39f676fb","Type":"ContainerDied","Data":"3b0c764670e6fa38e21238377b9639a692d0fb6d8cdbcd64876e2b2adea24c0e"}
Oct 30 20:35:19.237207 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:19.237181    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-84mb7"
Oct 30 20:35:19.333285 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:19.333261    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a46cc1e5-5fa0-46e2-9cee-fbac39f676fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a46cc1e5-5fa0-46e2-9cee-fbac39f676fb-maas-api-openshift-ca\") pod \"a46cc1e5-5fa0-46e2-9cee-fbac39f676fb\" (UID: \"a46cc1e5-5fa0-46e2-9cee-fbac39f676fb\") "
Oct 30 20:35:19.333409 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:19.333351    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a46cc1e5-5fa0-46e2-9cee-fbac39f676fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a46cc1e5-5fa0-46e2-9cee-fbac39f676fb-maas-api-xks-ca\") pod \"a46cc1e5-5fa0-46e2-9cee-fbac39f676fb\" (UID: \"a46cc1e5-5fa0-46e2-9cee-fbac39f676fb\") "
Oct 30 20:35:19.333409 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:19.333384    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a46cc1e5-5fa0-46e2-9cee-fbac39f676fb-kube-api-access-x6l7p\" (UniqueName: \"kubernetes.io/projected/a46cc1e5-5fa0-46e2-9cee-fbac39f676fb-kube-api-access-x6l7p\") pod \"a46cc1e5-5fa0-46e2-9cee-fbac39f676fb\" (UID: \"a46cc1e5-5fa0-46e2-9cee-fbac39f676fb\") "
Oct 30 20:35:19.333609 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:19.333588    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a46cc1e5-5fa0-46e2-9cee-fbac39f676fb-maas-api-openshift-ca" pod "a46cc1e5-5fa0-46e2-9cee-fbac39f676fb" (UID: "a46cc1e5-5fa0-46e2-9cee-fbac39f676fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:35:19.335474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:19.335439    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a46cc1e5-5fa0-46e2-9cee-fbac39f676fb-maas-api-xks-ca" pod "a46cc1e5-5fa0-46e2-9cee-fbac39f676fb" (UID: "a46cc1e5-5fa0-46e2-9cee-fbac39f676fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:35:19.335597 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:19.335558    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a46cc1e5-5fa0-46e2-9cee-fbac39f676fb-kube-api-access-x6l7p" pod "a46cc1e5-5fa0-46e2-9cee-fbac39f676fb" (UID: "a46cc1e5-5fa0-46e2-9cee-fbac39f676fb"). InnerVolumeSpecName "kube-api-access-x6l7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:19.434117 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:19.434089    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a46cc1e5-5fa0-46e2-9cee-fbac39f676fb-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:19.434117 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:19.434117    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a46cc1e5-5fa0-46e2-9cee-fbac39f676fb-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:19.434262 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:19.434127    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6l7p\" (UniqueName: \"kubernetes.io/projected/a46cc1e5-5fa0-46e2-9cee-fbac39f676fb-kube-api-access-x6l7p\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:20.108159 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:20.108134    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-84mb7"
Oct 30 20:35:20.108314 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:20.108173    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-84mb7" event={"ID":"a46cc1e5-5fa0-46e2-9cee-fbac39f676fb","Type":"ContainerDied","Data":"6993b3f55bff768dd52891e1f7dfd67fee677b648122e735fff7ee76f0b211ae"}
Oct 30 20:35:20.108314 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:20.108221    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6993b3f55bff768dd52891e1f7dfd67fee677b648122e735fff7ee76f0b211ae"
Oct 30 20:35:36.605113 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:35:36.605076    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:35:42.814277 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:42.814226    5958 scope.go:122] "RemoveContainer" containerID="60290760dceb9c7c8c1fc7a8f9c7c448b5d5d23221df08b8a3751bc4f079cbb7"
Oct 30 20:35:42.822957 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:42.822933    5958 scope.go:122] "RemoveContainer" containerID="f4fc9a0583d527d85732fc08715d2fef1544c8f85e331b9bbcd40e96924e3a49"
Oct 30 20:35:47.196533 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:47.196501    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-f2hzt"]
Oct 30 20:35:47.196929 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:47.196915    5958 cpu_manager.go:414] "containerMap: removing container" podUID="a46cc1e5-5fa0-46e2-9cee-fbac39f676fb" containerName="revoke-keys"
Oct 30 20:35:47.196975 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:47.196933    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a46cc1e5-5fa0-46e2-9cee-fbac39f676fb" containerName="revoke-keys"
Oct 30 20:35:47.197027 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:47.197018    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="a46cc1e5-5fa0-46e2-9cee-fbac39f676fb" containerName="revoke-keys"
Oct 30 20:35:47.201756 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:47.201736    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-f2hzt"
Oct 30 20:35:47.255310 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:47.255283    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:35:47.255495 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:47.255358    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:35:47.305166 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:47.305141    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:35:47.305397 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:47.305378    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:35:47.307178 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:47.307152    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-f2hzt"]
Oct 30 20:35:47.329231 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:47.329206    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/242fe0e8-092a-4b34-8d79-e6f111215826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-f2hzt\" (UID: \"242fe0e8-092a-4b34-8d79-e6f111215826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-f2hzt"
Oct 30 20:35:47.329344 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:47.329251    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/242fe0e8-092a-4b34-8d79-e6f111215826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-f2hzt\" (UID: \"242fe0e8-092a-4b34-8d79-e6f111215826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-f2hzt"
Oct 30 20:35:47.329399 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:47.329381    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8hx7\" (UniqueName: \"kubernetes.io/projected/242fe0e8-092a-4b34-8d79-e6f111215826-kube-api-access-z8hx7\") pod \"maas-api-revoke-sub-a4aef7d7-f2hzt\" (UID: \"242fe0e8-092a-4b34-8d79-e6f111215826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-f2hzt"
Oct 30 20:35:47.429825 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:47.429794    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/242fe0e8-092a-4b34-8d79-e6f111215826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-f2hzt\" (UID: \"242fe0e8-092a-4b34-8d79-e6f111215826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-f2hzt"
Oct 30 20:35:47.429962 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:47.429878    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8hx7\" (UniqueName: \"kubernetes.io/projected/242fe0e8-092a-4b34-8d79-e6f111215826-kube-api-access-z8hx7\") pod \"maas-api-revoke-sub-a4aef7d7-f2hzt\" (UID: \"242fe0e8-092a-4b34-8d79-e6f111215826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-f2hzt"
Oct 30 20:35:47.429962 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:47.429934    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/242fe0e8-092a-4b34-8d79-e6f111215826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-f2hzt\" (UID: \"242fe0e8-092a-4b34-8d79-e6f111215826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-f2hzt"
Oct 30 20:35:47.430493 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:47.430471    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/242fe0e8-092a-4b34-8d79-e6f111215826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-f2hzt\" (UID: \"242fe0e8-092a-4b34-8d79-e6f111215826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-f2hzt"
Oct 30 20:35:47.432224 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:47.432201    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/242fe0e8-092a-4b34-8d79-e6f111215826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-f2hzt\" (UID: \"242fe0e8-092a-4b34-8d79-e6f111215826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-f2hzt"
Oct 30 20:35:47.491496 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:47.491420    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8hx7\" (UniqueName: \"kubernetes.io/projected/242fe0e8-092a-4b34-8d79-e6f111215826-kube-api-access-z8hx7\") pod \"maas-api-revoke-sub-a4aef7d7-f2hzt\" (UID: \"242fe0e8-092a-4b34-8d79-e6f111215826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-f2hzt"
Oct 30 20:35:47.515255 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:47.515231    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-f2hzt"
Oct 30 20:35:47.899358 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:47.898242    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-f2hzt"]
Oct 30 20:35:48.346990 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:48.346954    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-f2hzt" event={"ID":"242fe0e8-092a-4b34-8d79-e6f111215826","Type":"ContainerStarted","Data":"0a70e687cbc4a77855a097489dfc4283b98f278917c1575411f2c4b4cb024ae9"}
Oct 30 20:35:48.347342 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:48.347016    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-f2hzt" event={"ID":"242fe0e8-092a-4b34-8d79-e6f111215826","Type":"ContainerStarted","Data":"d0672858da17d6808eb73f78e3269acc5d6979ec06d11915d05ddf3cf399c653"}
Oct 30 20:35:48.465200 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:48.465118    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-f2hzt" podStartSLOduration=1.465104432 podStartE2EDuration="1.465104432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:48.4635199 +0000 UTC m=+3187.195735289" watchObservedRunningTime="2026-10-30 20:35:48.465104432 +0000 UTC m=+3187.197319829"
Oct 30 20:35:49.353070 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:49.353031    5958 generic.go:356] "Generic (PLEG): container finished" podID="242fe0e8-092a-4b34-8d79-e6f111215826" containerID="0a70e687cbc4a77855a097489dfc4283b98f278917c1575411f2c4b4cb024ae9" exitCode=0
Oct 30 20:35:49.353070 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:49.353071    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-f2hzt" event={"ID":"242fe0e8-092a-4b34-8d79-e6f111215826","Type":"ContainerDied","Data":"0a70e687cbc4a77855a097489dfc4283b98f278917c1575411f2c4b4cb024ae9"}
Oct 30 20:35:50.487189 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:50.487162    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-f2hzt"
Oct 30 20:35:50.557416 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:50.557390    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/242fe0e8-092a-4b34-8d79-e6f111215826-kube-api-access-z8hx7\" (UniqueName: \"kubernetes.io/projected/242fe0e8-092a-4b34-8d79-e6f111215826-kube-api-access-z8hx7\") pod \"242fe0e8-092a-4b34-8d79-e6f111215826\" (UID: \"242fe0e8-092a-4b34-8d79-e6f111215826\") "
Oct 30 20:35:50.557521 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:50.557426    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/242fe0e8-092a-4b34-8d79-e6f111215826-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/242fe0e8-092a-4b34-8d79-e6f111215826-maas-api-openshift-ca\") pod \"242fe0e8-092a-4b34-8d79-e6f111215826\" (UID: \"242fe0e8-092a-4b34-8d79-e6f111215826\") "
Oct 30 20:35:50.557521 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:50.557470    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/242fe0e8-092a-4b34-8d79-e6f111215826-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/242fe0e8-092a-4b34-8d79-e6f111215826-maas-api-xks-ca\") pod \"242fe0e8-092a-4b34-8d79-e6f111215826\" (UID: \"242fe0e8-092a-4b34-8d79-e6f111215826\") "
Oct 30 20:35:50.557799 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:50.557777    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242fe0e8-092a-4b34-8d79-e6f111215826-maas-api-openshift-ca" pod "242fe0e8-092a-4b34-8d79-e6f111215826" (UID: "242fe0e8-092a-4b34-8d79-e6f111215826"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:35:50.559568 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:50.559550    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242fe0e8-092a-4b34-8d79-e6f111215826-kube-api-access-z8hx7" pod "242fe0e8-092a-4b34-8d79-e6f111215826" (UID: "242fe0e8-092a-4b34-8d79-e6f111215826"). InnerVolumeSpecName "kube-api-access-z8hx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:50.559680 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:50.559565    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/242fe0e8-092a-4b34-8d79-e6f111215826-maas-api-xks-ca" pod "242fe0e8-092a-4b34-8d79-e6f111215826" (UID: "242fe0e8-092a-4b34-8d79-e6f111215826"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:35:50.658665 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:50.658608    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8hx7\" (UniqueName: \"kubernetes.io/projected/242fe0e8-092a-4b34-8d79-e6f111215826-kube-api-access-z8hx7\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:50.658665 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:50.658631    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/242fe0e8-092a-4b34-8d79-e6f111215826-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:50.658665 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:50.658640    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/242fe0e8-092a-4b34-8d79-e6f111215826-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:35:51.364523 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:51.364492    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-f2hzt"
Oct 30 20:35:51.364723 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:51.364499    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-f2hzt" event={"ID":"242fe0e8-092a-4b34-8d79-e6f111215826","Type":"ContainerDied","Data":"d0672858da17d6808eb73f78e3269acc5d6979ec06d11915d05ddf3cf399c653"}
Oct 30 20:35:51.364723 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:35:51.364617    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0672858da17d6808eb73f78e3269acc5d6979ec06d11915d05ddf3cf399c653"
Oct 30 20:36:06.606904 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:36:06.606861    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:36:13.056991 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:13.056960    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b954773c-gjrcj"]
Oct 30 20:36:13.105728 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:13.105697    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b954773c-gjrcj"]
Oct 30 20:36:13.397013 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:13.396980    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b96a18b-6691-4463-88dd-933e4057f7b4" path="/var/lib/kubelet/pods/3b96a18b-6691-4463-88dd-933e4057f7b4/volumes"
Oct 30 20:36:17.762831 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:17.762802    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qtghc"]
Oct 30 20:36:17.763285 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:17.763270    5958 cpu_manager.go:414] "containerMap: removing container" podUID="242fe0e8-092a-4b34-8d79-e6f111215826" containerName="revoke-keys"
Oct 30 20:36:17.763337 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:17.763286    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="242fe0e8-092a-4b34-8d79-e6f111215826" containerName="revoke-keys"
Oct 30 20:36:17.763389 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:17.763379    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="242fe0e8-092a-4b34-8d79-e6f111215826" containerName="revoke-keys"
Oct 30 20:36:17.766267 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:17.766251    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qtghc"
Oct 30 20:36:17.794112 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:17.794087    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bbc4e41-d6cc-4a9e-8615-42924552ce70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-qtghc\" (UID: \"1bbc4e41-d6cc-4a9e-8615-42924552ce70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qtghc"
Oct 30 20:36:17.794241 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:17.794126    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89xh6\" (UniqueName: \"kubernetes.io/projected/1bbc4e41-d6cc-4a9e-8615-42924552ce70-kube-api-access-89xh6\") pod \"maas-api-revoke-sub-79805ca9-qtghc\" (UID: \"1bbc4e41-d6cc-4a9e-8615-42924552ce70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qtghc"
Oct 30 20:36:17.794241 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:17.794163    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bbc4e41-d6cc-4a9e-8615-42924552ce70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-qtghc\" (UID: \"1bbc4e41-d6cc-4a9e-8615-42924552ce70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qtghc"
Oct 30 20:36:17.868950 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:17.868922    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:36:17.869093 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:17.868951    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:36:17.869093 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:17.868974    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:36:17.869093 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:17.868984    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:36:17.870918 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:17.870900    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qtghc"]
Oct 30 20:36:17.895235 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:17.895213    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bbc4e41-d6cc-4a9e-8615-42924552ce70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-qtghc\" (UID: \"1bbc4e41-d6cc-4a9e-8615-42924552ce70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qtghc"
Oct 30 20:36:17.895351 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:17.895257    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89xh6\" (UniqueName: \"kubernetes.io/projected/1bbc4e41-d6cc-4a9e-8615-42924552ce70-kube-api-access-89xh6\") pod \"maas-api-revoke-sub-79805ca9-qtghc\" (UID: \"1bbc4e41-d6cc-4a9e-8615-42924552ce70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qtghc"
Oct 30 20:36:17.895351 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:17.895301    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bbc4e41-d6cc-4a9e-8615-42924552ce70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-qtghc\" (UID: \"1bbc4e41-d6cc-4a9e-8615-42924552ce70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qtghc"
Oct 30 20:36:17.895985 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:17.895967    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bbc4e41-d6cc-4a9e-8615-42924552ce70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-qtghc\" (UID: \"1bbc4e41-d6cc-4a9e-8615-42924552ce70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qtghc"
Oct 30 20:36:17.897633 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:17.897607    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bbc4e41-d6cc-4a9e-8615-42924552ce70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-qtghc\" (UID: \"1bbc4e41-d6cc-4a9e-8615-42924552ce70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qtghc"
Oct 30 20:36:17.956746 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:17.956723    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89xh6\" (UniqueName: \"kubernetes.io/projected/1bbc4e41-d6cc-4a9e-8615-42924552ce70-kube-api-access-89xh6\") pod \"maas-api-revoke-sub-79805ca9-qtghc\" (UID: \"1bbc4e41-d6cc-4a9e-8615-42924552ce70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qtghc"
Oct 30 20:36:18.078868 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:18.078847    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qtghc"
Oct 30 20:36:18.451296 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:18.451271    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qtghc"]
Oct 30 20:36:18.453481 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:36:18.453424    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bbc4e41_d6cc_4a9e_8615_42924552ce70.slice/crio-6b1f11959318083c59343e2351054bbf6338be067cf5d3365ac153d21afdea5e WatchSource:0}: Error finding container 6b1f11959318083c59343e2351054bbf6338be067cf5d3365ac153d21afdea5e: Status 404 returned error can't find the container with id 6b1f11959318083c59343e2351054bbf6338be067cf5d3365ac153d21afdea5e
Oct 30 20:36:18.534213 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:18.534191    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qtghc" event={"ID":"1bbc4e41-d6cc-4a9e-8615-42924552ce70","Type":"ContainerStarted","Data":"6b1f11959318083c59343e2351054bbf6338be067cf5d3365ac153d21afdea5e"}
Oct 30 20:36:19.541551 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:19.541515    5958 generic.go:356] "Generic (PLEG): container finished" podID="1bbc4e41-d6cc-4a9e-8615-42924552ce70" containerID="cb5fa0186b8346240bc68087553c3609c3b4578d546d5d2ef5e869bf1bd6d18a" exitCode=0
Oct 30 20:36:19.541940 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:19.541567    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qtghc" event={"ID":"1bbc4e41-d6cc-4a9e-8615-42924552ce70","Type":"ContainerDied","Data":"cb5fa0186b8346240bc68087553c3609c3b4578d546d5d2ef5e869bf1bd6d18a"}
Oct 30 20:36:20.698402 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:20.698374    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qtghc"
Oct 30 20:36:20.719323 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:20.719284    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bbc4e41-d6cc-4a9e-8615-42924552ce70-kube-api-access-89xh6\" (UniqueName: \"kubernetes.io/projected/1bbc4e41-d6cc-4a9e-8615-42924552ce70-kube-api-access-89xh6\") pod \"1bbc4e41-d6cc-4a9e-8615-42924552ce70\" (UID: \"1bbc4e41-d6cc-4a9e-8615-42924552ce70\") "
Oct 30 20:36:20.719480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:20.719424    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bbc4e41-d6cc-4a9e-8615-42924552ce70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bbc4e41-d6cc-4a9e-8615-42924552ce70-maas-api-xks-ca\") pod \"1bbc4e41-d6cc-4a9e-8615-42924552ce70\" (UID: \"1bbc4e41-d6cc-4a9e-8615-42924552ce70\") "
Oct 30 20:36:20.719554 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:20.719484    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bbc4e41-d6cc-4a9e-8615-42924552ce70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bbc4e41-d6cc-4a9e-8615-42924552ce70-maas-api-openshift-ca\") pod \"1bbc4e41-d6cc-4a9e-8615-42924552ce70\" (UID: \"1bbc4e41-d6cc-4a9e-8615-42924552ce70\") "
Oct 30 20:36:20.719834 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:20.719811    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bbc4e41-d6cc-4a9e-8615-42924552ce70-maas-api-openshift-ca" pod "1bbc4e41-d6cc-4a9e-8615-42924552ce70" (UID: "1bbc4e41-d6cc-4a9e-8615-42924552ce70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:36:20.721386 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:20.721360    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bbc4e41-d6cc-4a9e-8615-42924552ce70-kube-api-access-89xh6" pod "1bbc4e41-d6cc-4a9e-8615-42924552ce70" (UID: "1bbc4e41-d6cc-4a9e-8615-42924552ce70"). InnerVolumeSpecName "kube-api-access-89xh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:36:20.721514 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:20.721442    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bbc4e41-d6cc-4a9e-8615-42924552ce70-maas-api-xks-ca" pod "1bbc4e41-d6cc-4a9e-8615-42924552ce70" (UID: "1bbc4e41-d6cc-4a9e-8615-42924552ce70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:36:20.820169 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:20.820141    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89xh6\" (UniqueName: \"kubernetes.io/projected/1bbc4e41-d6cc-4a9e-8615-42924552ce70-kube-api-access-89xh6\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:20.820169 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:20.820166    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bbc4e41-d6cc-4a9e-8615-42924552ce70-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:20.820373 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:20.820176    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bbc4e41-d6cc-4a9e-8615-42924552ce70-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:21.553753 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:21.553721    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qtghc" event={"ID":"1bbc4e41-d6cc-4a9e-8615-42924552ce70","Type":"ContainerDied","Data":"6b1f11959318083c59343e2351054bbf6338be067cf5d3365ac153d21afdea5e"}
Oct 30 20:36:21.553753 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:21.553737    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qtghc"
Oct 30 20:36:21.553929 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:21.553769    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b1f11959318083c59343e2351054bbf6338be067cf5d3365ac153d21afdea5e"
Oct 30 20:36:36.604270 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:36:36.604231    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:36:41.661872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:41.661836    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-llr8j"]
Oct 30 20:36:41.662305 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:41.662290    5958 cpu_manager.go:414] "containerMap: removing container" podUID="1bbc4e41-d6cc-4a9e-8615-42924552ce70" containerName="revoke-keys"
Oct 30 20:36:41.662348 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:41.662306    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bbc4e41-d6cc-4a9e-8615-42924552ce70" containerName="revoke-keys"
Oct 30 20:36:41.662478 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:41.662439    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bbc4e41-d6cc-4a9e-8615-42924552ce70" containerName="revoke-keys"
Oct 30 20:36:41.665712 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:41.665690    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-llr8j"
Oct 30 20:36:41.718541 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:41.718511    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:36:41.718687 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:41.718669    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:36:41.748411 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:41.748381    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-llr8j\" (UID: \"3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-llr8j"
Oct 30 20:36:41.748579 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:41.748436    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq2w7\" (UniqueName: \"kubernetes.io/projected/3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e-kube-api-access-gq2w7\") pod \"maas-api-revoke-sub-0b7e8edb-llr8j\" (UID: \"3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-llr8j"
Oct 30 20:36:41.748653 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:41.748595    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-llr8j\" (UID: \"3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-llr8j"
Oct 30 20:36:41.768709 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:41.768686    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:36:41.768830 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:41.768754    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:36:41.770759 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:41.770740    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-llr8j"]
Oct 30 20:36:41.849821 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:41.849796    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-llr8j\" (UID: \"3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-llr8j"
Oct 30 20:36:41.849944 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:41.849841    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq2w7\" (UniqueName: \"kubernetes.io/projected/3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e-kube-api-access-gq2w7\") pod \"maas-api-revoke-sub-0b7e8edb-llr8j\" (UID: \"3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-llr8j"
Oct 30 20:36:41.849944 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:41.849873    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-llr8j\" (UID: \"3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-llr8j"
Oct 30 20:36:41.850639 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:41.850576    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-llr8j\" (UID: \"3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-llr8j"
Oct 30 20:36:41.852184 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:41.852162    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-llr8j\" (UID: \"3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-llr8j"
Oct 30 20:36:41.933977 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:41.933923    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq2w7\" (UniqueName: \"kubernetes.io/projected/3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e-kube-api-access-gq2w7\") pod \"maas-api-revoke-sub-0b7e8edb-llr8j\" (UID: \"3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-llr8j"
Oct 30 20:36:41.985744 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:41.985724    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-llr8j"
Oct 30 20:36:42.358028 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:42.358003    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-llr8j"]
Oct 30 20:36:42.360023 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:36:42.359991    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a68fcf8_61a4_4e04_8ef4_43b13ba10c9e.slice/crio-cc2d6c9e605cc469b5171c4fd11522508249bfb1e70bd5b27b232a8631f7090a WatchSource:0}: Error finding container cc2d6c9e605cc469b5171c4fd11522508249bfb1e70bd5b27b232a8631f7090a: Status 404 returned error can't find the container with id cc2d6c9e605cc469b5171c4fd11522508249bfb1e70bd5b27b232a8631f7090a
Oct 30 20:36:42.687726 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:42.687695    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-llr8j" event={"ID":"3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e","Type":"ContainerStarted","Data":"e3ee0d3c4c1eca4cbe7185311c6a1d19b2f4a9e1c9e9a748b89a16cc723f2dc4"}
Oct 30 20:36:42.688154 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:42.687747    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-llr8j" event={"ID":"3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e","Type":"ContainerStarted","Data":"cc2d6c9e605cc469b5171c4fd11522508249bfb1e70bd5b27b232a8631f7090a"}
Oct 30 20:36:42.806348 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:42.806303    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-llr8j" podStartSLOduration=1.8062898870000001 podStartE2EDuration="1.806289887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:36:42.804394235 +0000 UTC m=+3241.536609627" watchObservedRunningTime="2026-10-30 20:36:42.806289887 +0000 UTC m=+3241.538505304"
Oct 30 20:36:42.846421 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:42.846398    5958 scope.go:122] "RemoveContainer" containerID="de5f3742666b270344deb90f058f902c00545ee1b4121d0dd60f02415326aa4e"
Oct 30 20:36:42.855082 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:42.855059    5958 scope.go:122] "RemoveContainer" containerID="551cc56a5fb8fd5675c10da4b69324919ceaeb531d4dda12f9c0efdb1ee20962"
Oct 30 20:36:42.864820 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:42.864799    5958 scope.go:122] "RemoveContainer" containerID="26f209e87ddd7fc2784fea80f9c3f18bf1eed2316452c56ac24518bd41edc17e"
Oct 30 20:36:43.694154 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:43.694123    5958 generic.go:356] "Generic (PLEG): container finished" podID="3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e" containerID="e3ee0d3c4c1eca4cbe7185311c6a1d19b2f4a9e1c9e9a748b89a16cc723f2dc4" exitCode=0
Oct 30 20:36:43.694568 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:43.694198    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-llr8j" event={"ID":"3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e","Type":"ContainerDied","Data":"e3ee0d3c4c1eca4cbe7185311c6a1d19b2f4a9e1c9e9a748b89a16cc723f2dc4"}
Oct 30 20:36:44.827711 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:44.827690    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-llr8j"
Oct 30 20:36:44.977051 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:44.976977    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e-maas-api-xks-ca\") pod \"3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e\" (UID: \"3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e\") "
Oct 30 20:36:44.977206 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:44.977058    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e-maas-api-openshift-ca\") pod \"3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e\" (UID: \"3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e\") "
Oct 30 20:36:44.977206 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:44.977098    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e-kube-api-access-gq2w7\" (UniqueName: \"kubernetes.io/projected/3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e-kube-api-access-gq2w7\") pod \"3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e\" (UID: \"3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e\") "
Oct 30 20:36:44.977487 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:44.977438    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e-maas-api-openshift-ca" pod "3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e" (UID: "3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:36:44.977622 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:44.977542    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:44.979263 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:44.979223    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e-maas-api-xks-ca" pod "3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e" (UID: "3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:36:44.979392 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:44.979313    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e-kube-api-access-gq2w7" pod "3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e" (UID: "3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e"). InnerVolumeSpecName "kube-api-access-gq2w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:36:45.078712 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:45.078688    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq2w7\" (UniqueName: \"kubernetes.io/projected/3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e-kube-api-access-gq2w7\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:45.078712 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:45.078711    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:45.712184 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:45.712116    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-llr8j"
Oct 30 20:36:45.712184 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:45.712148    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-llr8j" event={"ID":"3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e","Type":"ContainerDied","Data":"cc2d6c9e605cc469b5171c4fd11522508249bfb1e70bd5b27b232a8631f7090a"}
Oct 30 20:36:45.712411 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:45.712220    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc2d6c9e605cc469b5171c4fd11522508249bfb1e70bd5b27b232a8631f7090a"
Oct 30 20:36:53.010618 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:53.010543    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ffr5b"]
Oct 30 20:36:53.011061 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:53.011044    5958 cpu_manager.go:414] "containerMap: removing container" podUID="3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e" containerName="revoke-keys"
Oct 30 20:36:53.011108 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:53.011065    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e" containerName="revoke-keys"
Oct 30 20:36:53.011143 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:53.011137    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e" containerName="revoke-keys"
Oct 30 20:36:53.014580 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:53.014559    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ffr5b"
Oct 30 20:36:53.042743 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:53.042709    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b57992-c273-4a90-98a2-89ba29b802dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ffr5b\" (UID: \"35b57992-c273-4a90-98a2-89ba29b802dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ffr5b"
Oct 30 20:36:53.042852 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:53.042777    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b57992-c273-4a90-98a2-89ba29b802dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ffr5b\" (UID: \"35b57992-c273-4a90-98a2-89ba29b802dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ffr5b"
Oct 30 20:36:53.042938 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:53.042895    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k86ss\" (UniqueName: \"kubernetes.io/projected/35b57992-c273-4a90-98a2-89ba29b802dc-kube-api-access-k86ss\") pod \"maas-api-revoke-sub-3d4156f1-ffr5b\" (UID: \"35b57992-c273-4a90-98a2-89ba29b802dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ffr5b"
Oct 30 20:36:53.075659 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:53.075626    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ffr5b"]
Oct 30 20:36:53.119166 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:53.119143    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:36:53.119166 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:53.119140    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:36:53.119349 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:53.119196    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:36:53.119349 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:53.119257    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:36:53.143764 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:53.143742    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k86ss\" (UniqueName: \"kubernetes.io/projected/35b57992-c273-4a90-98a2-89ba29b802dc-kube-api-access-k86ss\") pod \"maas-api-revoke-sub-3d4156f1-ffr5b\" (UID: \"35b57992-c273-4a90-98a2-89ba29b802dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ffr5b"
Oct 30 20:36:53.143900 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:53.143838    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b57992-c273-4a90-98a2-89ba29b802dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ffr5b\" (UID: \"35b57992-c273-4a90-98a2-89ba29b802dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ffr5b"
Oct 30 20:36:53.143969 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:53.143907    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b57992-c273-4a90-98a2-89ba29b802dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ffr5b\" (UID: \"35b57992-c273-4a90-98a2-89ba29b802dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ffr5b"
Oct 30 20:36:53.144574 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:53.144507    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b57992-c273-4a90-98a2-89ba29b802dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ffr5b\" (UID: \"35b57992-c273-4a90-98a2-89ba29b802dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ffr5b"
Oct 30 20:36:53.146275 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:53.146255    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b57992-c273-4a90-98a2-89ba29b802dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ffr5b\" (UID: \"35b57992-c273-4a90-98a2-89ba29b802dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ffr5b"
Oct 30 20:36:53.205109 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:53.205087    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k86ss\" (UniqueName: \"kubernetes.io/projected/35b57992-c273-4a90-98a2-89ba29b802dc-kube-api-access-k86ss\") pod \"maas-api-revoke-sub-3d4156f1-ffr5b\" (UID: \"35b57992-c273-4a90-98a2-89ba29b802dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ffr5b"
Oct 30 20:36:53.327367 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:53.327342    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ffr5b"
Oct 30 20:36:53.713901 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:53.713874    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ffr5b"]
Oct 30 20:36:53.716560 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:36:53.716533    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b57992_c273_4a90_98a2_89ba29b802dc.slice/crio-aa8451efcc754ad242ea73b60ee35cc0a9d556c531ed33a3587751ff796257fd WatchSource:0}: Error finding container aa8451efcc754ad242ea73b60ee35cc0a9d556c531ed33a3587751ff796257fd: Status 404 returned error can't find the container with id aa8451efcc754ad242ea73b60ee35cc0a9d556c531ed33a3587751ff796257fd
Oct 30 20:36:53.754345 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:53.754317    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ffr5b" event={"ID":"35b57992-c273-4a90-98a2-89ba29b802dc","Type":"ContainerStarted","Data":"aa8451efcc754ad242ea73b60ee35cc0a9d556c531ed33a3587751ff796257fd"}
Oct 30 20:36:54.800535 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:54.799226    5958 generic.go:356] "Generic (PLEG): container finished" podID="35b57992-c273-4a90-98a2-89ba29b802dc" containerID="c83af0b3c6e645378cdd0845cad0dd96248a94fe2a0e3f236752972d07a7bc03" exitCode=0
Oct 30 20:36:54.800535 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:54.799401    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ffr5b" event={"ID":"35b57992-c273-4a90-98a2-89ba29b802dc","Type":"ContainerDied","Data":"c83af0b3c6e645378cdd0845cad0dd96248a94fe2a0e3f236752972d07a7bc03"}
Oct 30 20:36:56.130091 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:56.130069    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ffr5b"
Oct 30 20:36:56.211797 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:56.211750    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35b57992-c273-4a90-98a2-89ba29b802dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b57992-c273-4a90-98a2-89ba29b802dc-maas-api-xks-ca\") pod \"35b57992-c273-4a90-98a2-89ba29b802dc\" (UID: \"35b57992-c273-4a90-98a2-89ba29b802dc\") "
Oct 30 20:36:56.212404 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:56.212381    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35b57992-c273-4a90-98a2-89ba29b802dc-kube-api-access-k86ss\" (UniqueName: \"kubernetes.io/projected/35b57992-c273-4a90-98a2-89ba29b802dc-kube-api-access-k86ss\") pod \"35b57992-c273-4a90-98a2-89ba29b802dc\" (UID: \"35b57992-c273-4a90-98a2-89ba29b802dc\") "
Oct 30 20:36:56.212841 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:56.212820    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35b57992-c273-4a90-98a2-89ba29b802dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b57992-c273-4a90-98a2-89ba29b802dc-maas-api-openshift-ca\") pod \"35b57992-c273-4a90-98a2-89ba29b802dc\" (UID: \"35b57992-c273-4a90-98a2-89ba29b802dc\") "
Oct 30 20:36:56.213425 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:56.213294    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b57992-c273-4a90-98a2-89ba29b802dc-maas-api-openshift-ca" pod "35b57992-c273-4a90-98a2-89ba29b802dc" (UID: "35b57992-c273-4a90-98a2-89ba29b802dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:36:56.213952 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:56.213932    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b57992-c273-4a90-98a2-89ba29b802dc-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:56.216879 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:56.216853    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b57992-c273-4a90-98a2-89ba29b802dc-kube-api-access-k86ss" pod "35b57992-c273-4a90-98a2-89ba29b802dc" (UID: "35b57992-c273-4a90-98a2-89ba29b802dc"). InnerVolumeSpecName "kube-api-access-k86ss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:36:56.216957 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:56.216944    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35b57992-c273-4a90-98a2-89ba29b802dc-maas-api-xks-ca" pod "35b57992-c273-4a90-98a2-89ba29b802dc" (UID: "35b57992-c273-4a90-98a2-89ba29b802dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:36:56.314937 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:56.314893    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b57992-c273-4a90-98a2-89ba29b802dc-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:56.314937 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:56.314932    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k86ss\" (UniqueName: \"kubernetes.io/projected/35b57992-c273-4a90-98a2-89ba29b802dc-kube-api-access-k86ss\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:36:56.816303 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:56.816275    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ffr5b"
Oct 30 20:36:56.816486 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:56.816314    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ffr5b" event={"ID":"35b57992-c273-4a90-98a2-89ba29b802dc","Type":"ContainerDied","Data":"aa8451efcc754ad242ea73b60ee35cc0a9d556c531ed33a3587751ff796257fd"}
Oct 30 20:36:56.816486 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:36:56.816360    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa8451efcc754ad242ea73b60ee35cc0a9d556c531ed33a3587751ff796257fd"
Oct 30 20:37:06.604933 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:37:06.604899    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:37:08.057361 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:08.057314    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nm9rf"]
Oct 30 20:37:08.107080 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:08.107042    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nm9rf"]
Oct 30 20:37:09.412927 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:09.412884    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5370ed05-f050-4c48-a623-17239dd5408d" path="/var/lib/kubelet/pods/5370ed05-f050-4c48-a623-17239dd5408d/volumes"
Oct 30 20:37:23.747033 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:23.746999    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ffr5b"]
Oct 30 20:37:23.796970 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:23.796941    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ffr5b"]
Oct 30 20:37:23.889038 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:23.889010    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w8sst"]
Oct 30 20:37:23.889462 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:23.889435    5958 cpu_manager.go:414] "containerMap: removing container" podUID="35b57992-c273-4a90-98a2-89ba29b802dc" containerName="revoke-keys"
Oct 30 20:37:23.889518 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:23.889466    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35b57992-c273-4a90-98a2-89ba29b802dc" containerName="revoke-keys"
Oct 30 20:37:23.889569 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:23.889559    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="35b57992-c273-4a90-98a2-89ba29b802dc" containerName="revoke-keys"
Oct 30 20:37:23.892748 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:23.892731    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w8sst"
Oct 30 20:37:23.944984 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:23.944959    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbc53eb0-2328-46bf-8065-8b952cbe5095-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-w8sst\" (UID: \"dbc53eb0-2328-46bf-8065-8b952cbe5095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w8sst"
Oct 30 20:37:23.945091 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:23.944993    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbc53eb0-2328-46bf-8065-8b952cbe5095-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-w8sst\" (UID: \"dbc53eb0-2328-46bf-8065-8b952cbe5095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w8sst"
Oct 30 20:37:23.945091 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:23.945023    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz68j\" (UniqueName: \"kubernetes.io/projected/dbc53eb0-2328-46bf-8065-8b952cbe5095-kube-api-access-pz68j\") pod \"maas-api-revoke-sub-3d4156f1-w8sst\" (UID: \"dbc53eb0-2328-46bf-8065-8b952cbe5095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w8sst"
Oct 30 20:37:23.949302 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:23.949281    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:37:23.949302 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:23.949290    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:37:23.955609 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:23.955595    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:37:23.957618 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:23.957603    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:37:24.015604 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:24.015502    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w8sst"]
Oct 30 20:37:24.048174 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:24.048040    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbc53eb0-2328-46bf-8065-8b952cbe5095-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-w8sst\" (UID: \"dbc53eb0-2328-46bf-8065-8b952cbe5095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w8sst"
Oct 30 20:37:24.048303 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:24.048215    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbc53eb0-2328-46bf-8065-8b952cbe5095-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-w8sst\" (UID: \"dbc53eb0-2328-46bf-8065-8b952cbe5095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w8sst"
Oct 30 20:37:24.048303 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:24.048296    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz68j\" (UniqueName: \"kubernetes.io/projected/dbc53eb0-2328-46bf-8065-8b952cbe5095-kube-api-access-pz68j\") pod \"maas-api-revoke-sub-3d4156f1-w8sst\" (UID: \"dbc53eb0-2328-46bf-8065-8b952cbe5095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w8sst"
Oct 30 20:37:24.049072 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:24.049045    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbc53eb0-2328-46bf-8065-8b952cbe5095-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-w8sst\" (UID: \"dbc53eb0-2328-46bf-8065-8b952cbe5095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w8sst"
Oct 30 20:37:24.054944 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:24.054920    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbc53eb0-2328-46bf-8065-8b952cbe5095-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-w8sst\" (UID: \"dbc53eb0-2328-46bf-8065-8b952cbe5095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w8sst"
Oct 30 20:37:24.111085 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:24.111063    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz68j\" (UniqueName: \"kubernetes.io/projected/dbc53eb0-2328-46bf-8065-8b952cbe5095-kube-api-access-pz68j\") pod \"maas-api-revoke-sub-3d4156f1-w8sst\" (UID: \"dbc53eb0-2328-46bf-8065-8b952cbe5095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w8sst"
Oct 30 20:37:24.205541 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:24.205519    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w8sst"
Oct 30 20:37:24.581844 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:24.581816    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w8sst"]
Oct 30 20:37:24.583394 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:37:24.583365    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc53eb0_2328_46bf_8065_8b952cbe5095.slice/crio-2e94d87b287612bc8f2a16c3f9f5f41275c9ad7db4d384bd7a6c429aed8fae45 WatchSource:0}: Error finding container 2e94d87b287612bc8f2a16c3f9f5f41275c9ad7db4d384bd7a6c429aed8fae45: Status 404 returned error can't find the container with id 2e94d87b287612bc8f2a16c3f9f5f41275c9ad7db4d384bd7a6c429aed8fae45
Oct 30 20:37:24.993985 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:24.993957    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w8sst" event={"ID":"dbc53eb0-2328-46bf-8065-8b952cbe5095","Type":"ContainerStarted","Data":"e23d298448d7962766c350d327a08da5762bbab48b6addd0c90a05c5bb20ef90"}
Oct 30 20:37:24.994315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:24.994016    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w8sst" event={"ID":"dbc53eb0-2328-46bf-8065-8b952cbe5095","Type":"ContainerStarted","Data":"2e94d87b287612bc8f2a16c3f9f5f41275c9ad7db4d384bd7a6c429aed8fae45"}
Oct 30 20:37:25.114079 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:25.114009    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w8sst" podStartSLOduration=2.113996992 podStartE2EDuration="2.113996992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:25.110179717 +0000 UTC m=+3283.842395107" watchObservedRunningTime="2026-10-30 20:37:25.113996992 +0000 UTC m=+3283.846212381"
Oct 30 20:37:25.396915 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:25.396848    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35b57992-c273-4a90-98a2-89ba29b802dc" path="/var/lib/kubelet/pods/35b57992-c273-4a90-98a2-89ba29b802dc/volumes"
Oct 30 20:37:26.000674 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:26.000639    5958 generic.go:356] "Generic (PLEG): container finished" podID="dbc53eb0-2328-46bf-8065-8b952cbe5095" containerID="e23d298448d7962766c350d327a08da5762bbab48b6addd0c90a05c5bb20ef90" exitCode=0
Oct 30 20:37:26.001015 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:26.000717    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w8sst" event={"ID":"dbc53eb0-2328-46bf-8065-8b952cbe5095","Type":"ContainerDied","Data":"e23d298448d7962766c350d327a08da5762bbab48b6addd0c90a05c5bb20ef90"}
Oct 30 20:37:27.134671 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:27.134638    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w8sst"
Oct 30 20:37:27.278671 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:27.278601    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbc53eb0-2328-46bf-8065-8b952cbe5095-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbc53eb0-2328-46bf-8065-8b952cbe5095-maas-api-openshift-ca\") pod \"dbc53eb0-2328-46bf-8065-8b952cbe5095\" (UID: \"dbc53eb0-2328-46bf-8065-8b952cbe5095\") "
Oct 30 20:37:27.278795 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:27.278697    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbc53eb0-2328-46bf-8065-8b952cbe5095-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbc53eb0-2328-46bf-8065-8b952cbe5095-maas-api-xks-ca\") pod \"dbc53eb0-2328-46bf-8065-8b952cbe5095\" (UID: \"dbc53eb0-2328-46bf-8065-8b952cbe5095\") "
Oct 30 20:37:27.278795 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:27.278729    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbc53eb0-2328-46bf-8065-8b952cbe5095-kube-api-access-pz68j\" (UniqueName: \"kubernetes.io/projected/dbc53eb0-2328-46bf-8065-8b952cbe5095-kube-api-access-pz68j\") pod \"dbc53eb0-2328-46bf-8065-8b952cbe5095\" (UID: \"dbc53eb0-2328-46bf-8065-8b952cbe5095\") "
Oct 30 20:37:27.279004 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:27.278978    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbc53eb0-2328-46bf-8065-8b952cbe5095-maas-api-openshift-ca" pod "dbc53eb0-2328-46bf-8065-8b952cbe5095" (UID: "dbc53eb0-2328-46bf-8065-8b952cbe5095"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:37:27.279149 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:27.279133    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbc53eb0-2328-46bf-8065-8b952cbe5095-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:37:27.280903 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:27.280881    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc53eb0-2328-46bf-8065-8b952cbe5095-kube-api-access-pz68j" pod "dbc53eb0-2328-46bf-8065-8b952cbe5095" (UID: "dbc53eb0-2328-46bf-8065-8b952cbe5095"). InnerVolumeSpecName "kube-api-access-pz68j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:37:27.281267 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:27.281249    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbc53eb0-2328-46bf-8065-8b952cbe5095-maas-api-xks-ca" pod "dbc53eb0-2328-46bf-8065-8b952cbe5095" (UID: "dbc53eb0-2328-46bf-8065-8b952cbe5095"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:37:27.379610 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:27.379585    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbc53eb0-2328-46bf-8065-8b952cbe5095-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:37:27.379610 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:27.379608    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz68j\" (UniqueName: \"kubernetes.io/projected/dbc53eb0-2328-46bf-8065-8b952cbe5095-kube-api-access-pz68j\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:37:28.012421 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:28.012396    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w8sst"
Oct 30 20:37:28.012593 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:28.012418    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w8sst" event={"ID":"dbc53eb0-2328-46bf-8065-8b952cbe5095","Type":"ContainerDied","Data":"2e94d87b287612bc8f2a16c3f9f5f41275c9ad7db4d384bd7a6c429aed8fae45"}
Oct 30 20:37:28.012593 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:28.012482    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e94d87b287612bc8f2a16c3f9f5f41275c9ad7db4d384bd7a6c429aed8fae45"
Oct 30 20:37:36.607876 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:37:36.607834    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:37:41.944517 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:41.944484    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 20:37:41.958420 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:41.958396    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 20:37:42.894676 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:42.894647    5958 scope.go:122] "RemoveContainer" containerID="a352cfd4c46d84ade575c03a11b5b3f98280d9e626dd76ca3342cbbb6731666b"
Oct 30 20:37:42.903229 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:42.903209    5958 scope.go:122] "RemoveContainer" containerID="d7279136a8f7e65923a7b127bd8500fd89cef80336eabec703698836efd971c6"
Oct 30 20:37:58.348510 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:58.348470    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w8sst"]
Oct 30 20:37:58.403040 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:58.403010    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w8sst"]
Oct 30 20:37:58.476791 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:58.476760    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sghxc"]
Oct 30 20:37:58.477275 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:58.477255    5958 cpu_manager.go:414] "containerMap: removing container" podUID="dbc53eb0-2328-46bf-8065-8b952cbe5095" containerName="revoke-keys"
Oct 30 20:37:58.477374 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:58.477277    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbc53eb0-2328-46bf-8065-8b952cbe5095" containerName="revoke-keys"
Oct 30 20:37:58.477433 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:58.477383    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbc53eb0-2328-46bf-8065-8b952cbe5095" containerName="revoke-keys"
Oct 30 20:37:58.481932 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:58.481913    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sghxc"
Oct 30 20:37:58.535995 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:58.535972    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:37:58.536107 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:58.536047    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:37:58.536182 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:58.536116    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:37:58.536182 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:58.536116    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:37:58.539204 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:58.539183    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sghxc"]
Oct 30 20:37:58.540863 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:58.540838    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a55b702-8c17-4d5e-af8e-01be7863c500-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-sghxc\" (UID: \"0a55b702-8c17-4d5e-af8e-01be7863c500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sghxc"
Oct 30 20:37:58.540982 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:58.540886    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a55b702-8c17-4d5e-af8e-01be7863c500-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-sghxc\" (UID: \"0a55b702-8c17-4d5e-af8e-01be7863c500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sghxc"
Oct 30 20:37:58.541047 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:58.540976    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6b6l\" (UniqueName: \"kubernetes.io/projected/0a55b702-8c17-4d5e-af8e-01be7863c500-kube-api-access-f6b6l\") pod \"maas-api-revoke-sub-3d4156f1-sghxc\" (UID: \"0a55b702-8c17-4d5e-af8e-01be7863c500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sghxc"
Oct 30 20:37:58.642010 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:58.641953    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a55b702-8c17-4d5e-af8e-01be7863c500-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-sghxc\" (UID: \"0a55b702-8c17-4d5e-af8e-01be7863c500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sghxc"
Oct 30 20:37:58.642120 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:58.642039    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6b6l\" (UniqueName: \"kubernetes.io/projected/0a55b702-8c17-4d5e-af8e-01be7863c500-kube-api-access-f6b6l\") pod \"maas-api-revoke-sub-3d4156f1-sghxc\" (UID: \"0a55b702-8c17-4d5e-af8e-01be7863c500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sghxc"
Oct 30 20:37:58.642190 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:58.642129    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a55b702-8c17-4d5e-af8e-01be7863c500-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-sghxc\" (UID: \"0a55b702-8c17-4d5e-af8e-01be7863c500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sghxc"
Oct 30 20:37:58.642688 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:58.642652    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a55b702-8c17-4d5e-af8e-01be7863c500-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-sghxc\" (UID: \"0a55b702-8c17-4d5e-af8e-01be7863c500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sghxc"
Oct 30 20:37:58.644571 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:58.644552    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a55b702-8c17-4d5e-af8e-01be7863c500-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-sghxc\" (UID: \"0a55b702-8c17-4d5e-af8e-01be7863c500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sghxc"
Oct 30 20:37:58.707207 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:58.707179    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6b6l\" (UniqueName: \"kubernetes.io/projected/0a55b702-8c17-4d5e-af8e-01be7863c500-kube-api-access-f6b6l\") pod \"maas-api-revoke-sub-3d4156f1-sghxc\" (UID: \"0a55b702-8c17-4d5e-af8e-01be7863c500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sghxc"
Oct 30 20:37:58.793848 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:58.793825    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sghxc"
Oct 30 20:37:59.186038 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:59.186011    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sghxc"]
Oct 30 20:37:59.188193 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:37:59.188166    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a55b702_8c17_4d5e_af8e_01be7863c500.slice/crio-303af1175b9b17b90573d317fac0ea51b7f13cbbb5abab30ddb424f50d60d6bf WatchSource:0}: Error finding container 303af1175b9b17b90573d317fac0ea51b7f13cbbb5abab30ddb424f50d60d6bf: Status 404 returned error can't find the container with id 303af1175b9b17b90573d317fac0ea51b7f13cbbb5abab30ddb424f50d60d6bf
Oct 30 20:37:59.397681 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:37:59.397659    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbc53eb0-2328-46bf-8065-8b952cbe5095" path="/var/lib/kubelet/pods/dbc53eb0-2328-46bf-8065-8b952cbe5095/volumes"
Oct 30 20:38:00.184019 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:00.183989    5958 generic.go:356] "Generic (PLEG): container finished" podID="0a55b702-8c17-4d5e-af8e-01be7863c500" containerID="c595c7e95119cac3a240fa2e526a44027ae39546b9d64fda3ea77c6f6612c4f3" exitCode=0
Oct 30 20:38:00.184149 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:00.184078    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sghxc" event={"ID":"0a55b702-8c17-4d5e-af8e-01be7863c500","Type":"ContainerDied","Data":"c595c7e95119cac3a240fa2e526a44027ae39546b9d64fda3ea77c6f6612c4f3"}
Oct 30 20:38:00.184149 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:00.184135    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sghxc" event={"ID":"0a55b702-8c17-4d5e-af8e-01be7863c500","Type":"ContainerStarted","Data":"303af1175b9b17b90573d317fac0ea51b7f13cbbb5abab30ddb424f50d60d6bf"}
Oct 30 20:38:01.318302 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:01.318278    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sghxc"
Oct 30 20:38:01.368115 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:01.368088    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a55b702-8c17-4d5e-af8e-01be7863c500-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a55b702-8c17-4d5e-af8e-01be7863c500-maas-api-xks-ca\") pod \"0a55b702-8c17-4d5e-af8e-01be7863c500\" (UID: \"0a55b702-8c17-4d5e-af8e-01be7863c500\") "
Oct 30 20:38:01.368229 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:01.368177    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a55b702-8c17-4d5e-af8e-01be7863c500-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a55b702-8c17-4d5e-af8e-01be7863c500-maas-api-openshift-ca\") pod \"0a55b702-8c17-4d5e-af8e-01be7863c500\" (UID: \"0a55b702-8c17-4d5e-af8e-01be7863c500\") "
Oct 30 20:38:01.368269 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:01.368244    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a55b702-8c17-4d5e-af8e-01be7863c500-kube-api-access-f6b6l\" (UniqueName: \"kubernetes.io/projected/0a55b702-8c17-4d5e-af8e-01be7863c500-kube-api-access-f6b6l\") pod \"0a55b702-8c17-4d5e-af8e-01be7863c500\" (UID: \"0a55b702-8c17-4d5e-af8e-01be7863c500\") "
Oct 30 20:38:01.368527 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:01.368505    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a55b702-8c17-4d5e-af8e-01be7863c500-maas-api-openshift-ca" pod "0a55b702-8c17-4d5e-af8e-01be7863c500" (UID: "0a55b702-8c17-4d5e-af8e-01be7863c500"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:38:01.370262 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:01.370238    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a55b702-8c17-4d5e-af8e-01be7863c500-maas-api-xks-ca" pod "0a55b702-8c17-4d5e-af8e-01be7863c500" (UID: "0a55b702-8c17-4d5e-af8e-01be7863c500"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:38:01.370351 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:01.370314    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a55b702-8c17-4d5e-af8e-01be7863c500-kube-api-access-f6b6l" pod "0a55b702-8c17-4d5e-af8e-01be7863c500" (UID: "0a55b702-8c17-4d5e-af8e-01be7863c500"). InnerVolumeSpecName "kube-api-access-f6b6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:38:01.469151 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:01.469077    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6b6l\" (UniqueName: \"kubernetes.io/projected/0a55b702-8c17-4d5e-af8e-01be7863c500-kube-api-access-f6b6l\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:38:01.469151 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:01.469103    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a55b702-8c17-4d5e-af8e-01be7863c500-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:38:01.469151 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:01.469112    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a55b702-8c17-4d5e-af8e-01be7863c500-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:38:02.197397 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:02.197368    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sghxc"
Oct 30 20:38:02.197588 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:02.197399    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sghxc" event={"ID":"0a55b702-8c17-4d5e-af8e-01be7863c500","Type":"ContainerDied","Data":"303af1175b9b17b90573d317fac0ea51b7f13cbbb5abab30ddb424f50d60d6bf"}
Oct 30 20:38:02.197588 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:02.197468    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="303af1175b9b17b90573d317fac0ea51b7f13cbbb5abab30ddb424f50d60d6bf"
Oct 30 20:38:06.604840 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:38:06.604805    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:38:08.058324 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:08.058284    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vdpxl"]
Oct 30 20:38:08.108085 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:08.108053    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vdpxl"]
Oct 30 20:38:09.397133 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:09.397095    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="837b36db-1b4d-46b5-85f0-2132c326f5e8" path="/var/lib/kubelet/pods/837b36db-1b4d-46b5-85f0-2132c326f5e8/volumes"
Oct 30 20:38:31.342162 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.338988    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"]
Oct 30 20:38:31.342162 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.339564    5958 cpu_manager.go:414] "containerMap: removing container" podUID="0a55b702-8c17-4d5e-af8e-01be7863c500" containerName="revoke-keys"
Oct 30 20:38:31.342162 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.339584    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a55b702-8c17-4d5e-af8e-01be7863c500" containerName="revoke-keys"
Oct 30 20:38:31.342162 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.339721    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a55b702-8c17-4d5e-af8e-01be7863c500" containerName="revoke-keys"
Oct 30 20:38:31.343672 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.343646    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.400253 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.400227    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-b2cfa0-openshift-default-dockercfg-2cxzx\""
Oct 30 20:38:31.449961 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.449932    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"]
Oct 30 20:38:31.514738 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.514703    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-credential-socket\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.514897 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.514771    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-workload-certs\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.514897 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.514827    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-token\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.514897 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.514866    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-envoy\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.515006 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.514945    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-podinfo\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.515006 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.514999    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsfmf\" (UniqueName: \"kubernetes.io/projected/4b333509-55f1-4d62-8bdc-154d2b47cc4b-kube-api-access-qsfmf\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.515107 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.515084    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-data\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.515142 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.515124    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-workload-socket\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.515190 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.515176    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istiod-ca-cert\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.615848 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.615780    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsfmf\" (UniqueName: \"kubernetes.io/projected/4b333509-55f1-4d62-8bdc-154d2b47cc4b-kube-api-access-qsfmf\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.615848 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.615827    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-data\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.616142 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.615853    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-workload-socket\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.616142 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.615886    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istiod-ca-cert\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.616142 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.615925    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-credential-socket\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.616142 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.615986    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-workload-certs\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.616142 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.616007    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-token\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.616142 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.616034    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-envoy\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.616142 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.616076    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-podinfo\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.616582 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.616228    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-data\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.616582 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.616308    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-workload-socket\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.616582 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.616549    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-credential-socket\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.616844 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.616821    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istiod-ca-cert\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.616915 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.616889    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-workload-certs\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.618587 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.618563    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-envoy\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.618885 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.618868    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-podinfo\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.681095 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.681050    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsfmf\" (UniqueName: \"kubernetes.io/projected/4b333509-55f1-4d62-8bdc-154d2b47cc4b-kube-api-access-qsfmf\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.681602 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.681580    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-token\") pod \"e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") " pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:31.964192 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:31.964129    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:32.398059 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:38:32.398022    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b333509_55f1_4d62_8bdc_154d2b47cc4b.slice/crio-1e94b72174decb8051c718424dff4200927719b358e2d3f5eeaa94fcb3db89a6 WatchSource:0}: Error finding container 1e94b72174decb8051c718424dff4200927719b358e2d3f5eeaa94fcb3db89a6: Status 404 returned error can't find the container with id 1e94b72174decb8051c718424dff4200927719b358e2d3f5eeaa94fcb3db89a6
Oct 30 20:38:32.399025 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:32.398998    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"]
Oct 30 20:38:32.400494 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:32.400446    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:38:32.400565 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:32.400525    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:38:32.400565 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:32.400551    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:38:33.361183 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:33.361142    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm" event={"ID":"4b333509-55f1-4d62-8bdc-154d2b47cc4b","Type":"ContainerStarted","Data":"fd60a6f19f173dd2c48d40e42114a1107b138893b0093cdf453ce31ba1165f6c"}
Oct 30 20:38:33.361357 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:33.361231    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm" event={"ID":"4b333509-55f1-4d62-8bdc-154d2b47cc4b","Type":"ContainerStarted","Data":"1e94b72174decb8051c718424dff4200927719b358e2d3f5eeaa94fcb3db89a6"}
Oct 30 20:38:33.585271 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:33.585211    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm" podStartSLOduration=2.585160369 podStartE2EDuration="2.585160369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:33.534971873 +0000 UTC m=+3352.267187263" watchObservedRunningTime="2026-10-30 20:38:33.585160369 +0000 UTC m=+3352.317375763"
Oct 30 20:38:33.965181 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:33.965147    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:33.969956 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:33.969932    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:34.366167 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:34.366135    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:34.367036 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:34.367012    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:38:36.604590 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:38:36.604549    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:38:40.797949 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:40.797920    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv"]
Oct 30 20:38:40.802096 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:40.802072    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv"
Oct 30 20:38:40.857359 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:40.857329    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-8144f265-dockercfg-ph9lt\""
Oct 30 20:38:40.901111 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:40.901076    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b47z7\" (UniqueName: \"kubernetes.io/projected/7419f2cd-8447-48ed-9005-2014f48336a1-kube-api-access-b47z7\") pod \"payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv\" (UID: \"7419f2cd-8447-48ed-9005-2014f48336a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv"
Oct 30 20:38:40.901259 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:40.901234    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7419f2cd-8447-48ed-9005-2014f48336a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv\" (UID: \"7419f2cd-8447-48ed-9005-2014f48336a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv"
Oct 30 20:38:40.907208 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:40.907186    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-b2cfa0\""
Oct 30 20:38:40.909221 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:40.909197    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv"]
Oct 30 20:38:41.001871 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:41.001844    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7419f2cd-8447-48ed-9005-2014f48336a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv\" (UID: \"7419f2cd-8447-48ed-9005-2014f48336a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv"
Oct 30 20:38:41.001985 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:41.001930    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b47z7\" (UniqueName: \"kubernetes.io/projected/7419f2cd-8447-48ed-9005-2014f48336a1-kube-api-access-b47z7\") pod \"payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv\" (UID: \"7419f2cd-8447-48ed-9005-2014f48336a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv"
Oct 30 20:38:41.002536 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:41.002519    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7419f2cd-8447-48ed-9005-2014f48336a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv\" (UID: \"7419f2cd-8447-48ed-9005-2014f48336a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv"
Oct 30 20:38:41.064110 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:41.064088    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b47z7\" (UniqueName: \"kubernetes.io/projected/7419f2cd-8447-48ed-9005-2014f48336a1-kube-api-access-b47z7\") pod \"payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv\" (UID: \"7419f2cd-8447-48ed-9005-2014f48336a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv"
Oct 30 20:38:41.115136 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:41.115116    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv"
Oct 30 20:38:41.488576 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:41.488550    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv"]
Oct 30 20:38:41.490821 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:38:41.490794    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7419f2cd_8447_48ed_9005_2014f48336a1.slice/crio-99950da8f8a5813b7fa4016d9d637db3b7d78c68f60ff85163189f66b6588240 WatchSource:0}: Error finding container 99950da8f8a5813b7fa4016d9d637db3b7d78c68f60ff85163189f66b6588240: Status 404 returned error can't find the container with id 99950da8f8a5813b7fa4016d9d637db3b7d78c68f60ff85163189f66b6588240
Oct 30 20:38:42.066560 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.066531    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4zq44"]
Oct 30 20:38:42.115819 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.115791    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4zq44"]
Oct 30 20:38:42.412518 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.412405    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv" event={"ID":"7419f2cd-8447-48ed-9005-2014f48336a1","Type":"ContainerStarted","Data":"1ff4d95bfdd02dd5b8af05c20b2952751b6db13c16f68e606087d7963784b631"}
Oct 30 20:38:42.412518 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.412468    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv" event={"ID":"7419f2cd-8447-48ed-9005-2014f48336a1","Type":"ContainerStarted","Data":"99950da8f8a5813b7fa4016d9d637db3b7d78c68f60ff85163189f66b6588240"}
Oct 30 20:38:42.412692 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.412625    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv"
Oct 30 20:38:42.531512 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.531463    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv" podStartSLOduration=2.531436619 podStartE2EDuration="2.531436619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:42.529338493 +0000 UTC m=+3361.261553883" watchObservedRunningTime="2026-10-30 20:38:42.531436619 +0000 UTC m=+3361.263652009"
Oct 30 20:38:42.584788 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.584757    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj"]
Oct 30 20:38:42.588578 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.588559    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj"
Oct 30 20:38:42.619593 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.619570    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e41a65a-4901-48f6-b3af-78dfe213c601-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj\" (UID: \"2e41a65a-4901-48f6-b3af-78dfe213c601\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj"
Oct 30 20:38:42.619706 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.619603    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e41a65a-4901-48f6-b3af-78dfe213c601-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj\" (UID: \"2e41a65a-4901-48f6-b3af-78dfe213c601\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj"
Oct 30 20:38:42.619706 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.619642    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e41a65a-4901-48f6-b3af-78dfe213c601-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj\" (UID: \"2e41a65a-4901-48f6-b3af-78dfe213c601\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj"
Oct 30 20:38:42.619803 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.619786    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prmj4\" (UniqueName: \"kubernetes.io/projected/2e41a65a-4901-48f6-b3af-78dfe213c601-kube-api-access-prmj4\") pod \"maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj\" (UID: \"2e41a65a-4901-48f6-b3af-78dfe213c601\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj"
Oct 30 20:38:42.642631 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.642609    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xtx52\""
Oct 30 20:38:42.642913 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.642887    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:38:42.643559 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.643522    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-b2cfa0-serving-cert\""
Oct 30 20:38:42.643559 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.643538    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:38:42.643709 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.643556    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 20:38:42.693875 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.693801    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj"]
Oct 30 20:38:42.721263 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.721233    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prmj4\" (UniqueName: \"kubernetes.io/projected/2e41a65a-4901-48f6-b3af-78dfe213c601-kube-api-access-prmj4\") pod \"maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj\" (UID: \"2e41a65a-4901-48f6-b3af-78dfe213c601\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj"
Oct 30 20:38:42.721437 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.721279    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e41a65a-4901-48f6-b3af-78dfe213c601-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj\" (UID: \"2e41a65a-4901-48f6-b3af-78dfe213c601\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj"
Oct 30 20:38:42.721437 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.721310    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e41a65a-4901-48f6-b3af-78dfe213c601-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj\" (UID: \"2e41a65a-4901-48f6-b3af-78dfe213c601\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj"
Oct 30 20:38:42.721437 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.721356    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e41a65a-4901-48f6-b3af-78dfe213c601-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj\" (UID: \"2e41a65a-4901-48f6-b3af-78dfe213c601\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj"
Oct 30 20:38:42.721672 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:38:42.721607    5958 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-serving-cert: secret "maas-api-e2e-worker-single-main-b2cfa0-serving-cert" not found
Oct 30 20:38:42.721672 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:38:42.721670    5958 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e41a65a-4901-48f6-b3af-78dfe213c601-maas-api-tls podName:2e41a65a-4901-48f6-b3af-78dfe213c601 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:43.221652532 +0000 UTC m=+3361.953867902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2e41a65a-4901-48f6-b3af-78dfe213c601-maas-api-tls") pod "maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj" (UID: "2e41a65a-4901-48f6-b3af-78dfe213c601") : secret "maas-api-e2e-worker-single-main-b2cfa0-serving-cert" not found
Oct 30 20:38:42.722242 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.722213    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e41a65a-4901-48f6-b3af-78dfe213c601-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj\" (UID: \"2e41a65a-4901-48f6-b3af-78dfe213c601\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj"
Oct 30 20:38:42.724031 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.724002    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e41a65a-4901-48f6-b3af-78dfe213c601-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj\" (UID: \"2e41a65a-4901-48f6-b3af-78dfe213c601\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj"
Oct 30 20:38:42.785331 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.785308    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prmj4\" (UniqueName: \"kubernetes.io/projected/2e41a65a-4901-48f6-b3af-78dfe213c601-kube-api-access-prmj4\") pod \"maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj\" (UID: \"2e41a65a-4901-48f6-b3af-78dfe213c601\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj"
Oct 30 20:38:42.927912 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:42.927879    5958 scope.go:122] "RemoveContainer" containerID="beec1fa9102c26c7900a37f93856dee16d256b8ee3835f653df7dee4d785b69c"
Oct 30 20:38:43.225171 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:43.225095    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e41a65a-4901-48f6-b3af-78dfe213c601-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj\" (UID: \"2e41a65a-4901-48f6-b3af-78dfe213c601\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj"
Oct 30 20:38:43.227598 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:43.227571    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e41a65a-4901-48f6-b3af-78dfe213c601-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj\" (UID: \"2e41a65a-4901-48f6-b3af-78dfe213c601\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj"
Oct 30 20:38:43.397494 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:43.397443    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b2d7d1c-da0c-449d-b351-b6668a7d4e11" path="/var/lib/kubelet/pods/2b2d7d1c-da0c-449d-b351-b6668a7d4e11/volumes"
Oct 30 20:38:43.501614 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:43.501561    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj"
Oct 30 20:38:43.876432 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:43.876406    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj"]
Oct 30 20:38:43.881425 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:38:43.881381    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e41a65a_4901_48f6_b3af_78dfe213c601.slice/crio-28c533f66db08e9ab2d83b939e44473d02ade549ea9505ccf680efb47f920f7e WatchSource:0}: Error finding container 28c533f66db08e9ab2d83b939e44473d02ade549ea9505ccf680efb47f920f7e: Status 404 returned error can't find the container with id 28c533f66db08e9ab2d83b939e44473d02ade549ea9505ccf680efb47f920f7e
Oct 30 20:38:43.882838 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:43.882815    5958 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:38:44.424425 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:44.424393    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj" event={"ID":"2e41a65a-4901-48f6-b3af-78dfe213c601","Type":"ContainerStarted","Data":"28c533f66db08e9ab2d83b939e44473d02ade549ea9505ccf680efb47f920f7e"}
Oct 30 20:38:45.437961 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:45.437921    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj" event={"ID":"2e41a65a-4901-48f6-b3af-78dfe213c601","Type":"ContainerStarted","Data":"fccfc90067e5670504d2863baa8e9f90b7a954b37eb4a97d14c910198ef45f2d"}
Oct 30 20:38:45.438356 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:45.438020    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj"
Oct 30 20:38:45.560270 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:45.560221    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj" podStartSLOduration=2.665637523 podStartE2EDuration="3.560208579s" totalImagesPullingTime="894.571056ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:45.559371358 +0000 UTC m=+3364.291586748" watchObservedRunningTime="2026-10-30 20:38:45.560208579 +0000 UTC m=+3364.292423968"
Oct 30 20:38:51.461987 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:51.461952    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj"
Oct 30 20:38:53.421365 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:53.421337    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv"
Oct 30 20:38:55.061012 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:55.060969    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q7g68"]
Oct 30 20:38:55.109638 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:55.109610    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q7g68"]
Oct 30 20:38:55.403275 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:38:55.403234    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6517cb79-ec8b-4d28-bd56-e649e71fd225" path="/var/lib/kubelet/pods/6517cb79-ec8b-4d28-bd56-e649e71fd225/volumes"
Oct 30 20:39:00.518016 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:00.517978    5958 generic.go:356] "Generic (PLEG): container finished" podID="4b333509-55f1-4d62-8bdc-154d2b47cc4b" containerID="fd60a6f19f173dd2c48d40e42114a1107b138893b0093cdf453ce31ba1165f6c" exitCode=0
Oct 30 20:39:00.518431 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:00.518048    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm" event={"ID":"4b333509-55f1-4d62-8bdc-154d2b47cc4b","Type":"ContainerDied","Data":"fd60a6f19f173dd2c48d40e42114a1107b138893b0093cdf453ce31ba1165f6c"}
Oct 30 20:39:00.518431 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:00.518371    5958 scope.go:122] "RemoveContainer" containerID="fd60a6f19f173dd2c48d40e42114a1107b138893b0093cdf453ce31ba1165f6c"
Oct 30 20:39:00.519438 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:00.519409    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:39:00.519525 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:00.519496    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:39:00.519567 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:00.519527    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:39:01.525354 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:01.525308    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm" event={"ID":"4b333509-55f1-4d62-8bdc-154d2b47cc4b","Type":"ContainerStarted","Data":"9f7bb05b0773df1f5a93e2a159d2e1289809c329b64b269116bf39755209948f"}
Oct 30 20:39:01.964894 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:01.964861    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:39:01.966080 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:01.966047    5958 patch_prober.go:29] interesting pod/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 30 20:39:01.966202 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:01.966093    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm" podUID="4b333509-55f1-4d62-8bdc-154d2b47cc4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 30 20:39:02.964729 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:02.964695    5958 patch_prober.go:29] interesting pod/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 30 20:39:02.965220 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:02.964746    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm" podUID="4b333509-55f1-4d62-8bdc-154d2b47cc4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 30 20:39:03.964788 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:03.964743    5958 patch_prober.go:29] interesting pod/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 30 20:39:03.965278 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:03.964813    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm" podUID="4b333509-55f1-4d62-8bdc-154d2b47cc4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 30 20:39:04.366481 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:04.366416    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:39:04.965118 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:04.965077    5958 patch_prober.go:29] interesting pod/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 30 20:39:04.965644 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:04.965153    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm" podUID="4b333509-55f1-4d62-8bdc-154d2b47cc4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 30 20:39:05.965087 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:05.965049    5958 patch_prober.go:29] interesting pod/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 30 20:39:05.965272 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:05.965122    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm" podUID="4b333509-55f1-4d62-8bdc-154d2b47cc4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 30 20:39:06.605553 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:39:06.605518    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:39:06.968406 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:06.968331    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:39:06.969509 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:06.969485    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:39:34.056726 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:34.056688    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s84lt"]
Oct 30 20:39:34.105585 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:34.105549    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-s84lt"]
Oct 30 20:39:35.397150 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:35.397112    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4427003a-268c-42e4-9dc8-6ba925c73504" path="/var/lib/kubelet/pods/4427003a-268c-42e4-9dc8-6ba925c73504/volumes"
Oct 30 20:39:36.604401 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:39:36.604366    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:39:42.950550 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:42.950520    5958 scope.go:122] "RemoveContainer" containerID="164c30d28eafffe277f3372f6403de140df823929e8455648e090af1dd915494"
Oct 30 20:39:42.959499 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:42.959481    5958 scope.go:122] "RemoveContainer" containerID="7efac8c5f3a17d5f69005be6778c918156d710494c76a05ea239e6f345eebfcc"
Oct 30 20:39:44.053135 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:44.053102    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rpw7c"]
Oct 30 20:39:44.102803 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:44.102775    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rpw7c"]
Oct 30 20:39:45.399036 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:39:45.398970    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ad13ff5-8499-4767-bd69-b4f5d4131188" path="/var/lib/kubelet/pods/0ad13ff5-8499-4767-bd69-b4f5d4131188/volumes"
Oct 30 20:40:00.853863 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:00.853759    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv"]
Oct 30 20:40:02.180089 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:02.180053    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj"]
Oct 30 20:40:02.180580 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:02.180276    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj" podUID="2e41a65a-4901-48f6-b3af-78dfe213c601" containerName="maas-api" containerID="cri-o://fccfc90067e5670504d2863baa8e9f90b7a954b37eb4a97d14c910198ef45f2d" gracePeriod=30
Oct 30 20:40:03.436003 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:03.435976    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj"
Oct 30 20:40:03.518106 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:03.518047    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e41a65a-4901-48f6-b3af-78dfe213c601-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e41a65a-4901-48f6-b3af-78dfe213c601-metrics-certs\") pod \"2e41a65a-4901-48f6-b3af-78dfe213c601\" (UID: \"2e41a65a-4901-48f6-b3af-78dfe213c601\") "
Oct 30 20:40:03.518232 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:03.518164    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e41a65a-4901-48f6-b3af-78dfe213c601-kube-api-access-prmj4\" (UniqueName: \"kubernetes.io/projected/2e41a65a-4901-48f6-b3af-78dfe213c601-kube-api-access-prmj4\") pod \"2e41a65a-4901-48f6-b3af-78dfe213c601\" (UID: \"2e41a65a-4901-48f6-b3af-78dfe213c601\") "
Oct 30 20:40:03.518232 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:03.518191    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e41a65a-4901-48f6-b3af-78dfe213c601-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e41a65a-4901-48f6-b3af-78dfe213c601-openshift-service-ca\") pod \"2e41a65a-4901-48f6-b3af-78dfe213c601\" (UID: \"2e41a65a-4901-48f6-b3af-78dfe213c601\") "
Oct 30 20:40:03.518363 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:03.518280    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e41a65a-4901-48f6-b3af-78dfe213c601-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e41a65a-4901-48f6-b3af-78dfe213c601-maas-api-tls\") pod \"2e41a65a-4901-48f6-b3af-78dfe213c601\" (UID: \"2e41a65a-4901-48f6-b3af-78dfe213c601\") "
Oct 30 20:40:03.518520 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:03.518500    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e41a65a-4901-48f6-b3af-78dfe213c601-openshift-service-ca" pod "2e41a65a-4901-48f6-b3af-78dfe213c601" (UID: "2e41a65a-4901-48f6-b3af-78dfe213c601"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:40:03.518733 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:03.518715    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e41a65a-4901-48f6-b3af-78dfe213c601-openshift-service-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:03.520133 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:03.520110    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e41a65a-4901-48f6-b3af-78dfe213c601-metrics-certs" pod "2e41a65a-4901-48f6-b3af-78dfe213c601" (UID: "2e41a65a-4901-48f6-b3af-78dfe213c601"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:40:03.520362 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:03.520329    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e41a65a-4901-48f6-b3af-78dfe213c601-kube-api-access-prmj4" pod "2e41a65a-4901-48f6-b3af-78dfe213c601" (UID: "2e41a65a-4901-48f6-b3af-78dfe213c601"). InnerVolumeSpecName "kube-api-access-prmj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:40:03.520362 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:03.520341    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e41a65a-4901-48f6-b3af-78dfe213c601-maas-api-tls" pod "2e41a65a-4901-48f6-b3af-78dfe213c601" (UID: "2e41a65a-4901-48f6-b3af-78dfe213c601"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:40:03.619934 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:03.619908    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prmj4\" (UniqueName: \"kubernetes.io/projected/2e41a65a-4901-48f6-b3af-78dfe213c601-kube-api-access-prmj4\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:03.619934 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:03.619935    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e41a65a-4901-48f6-b3af-78dfe213c601-maas-api-tls\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:03.620095 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:03.619946    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e41a65a-4901-48f6-b3af-78dfe213c601-metrics-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:03.870243 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:03.870216    5958 generic.go:356] "Generic (PLEG): container finished" podID="2e41a65a-4901-48f6-b3af-78dfe213c601" containerID="fccfc90067e5670504d2863baa8e9f90b7a954b37eb4a97d14c910198ef45f2d" exitCode=0
Oct 30 20:40:03.870396 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:03.870265    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj"
Oct 30 20:40:03.870396 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:03.870304    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj" event={"ID":"2e41a65a-4901-48f6-b3af-78dfe213c601","Type":"ContainerDied","Data":"fccfc90067e5670504d2863baa8e9f90b7a954b37eb4a97d14c910198ef45f2d"}
Oct 30 20:40:03.870396 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:03.870357    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj" event={"ID":"2e41a65a-4901-48f6-b3af-78dfe213c601","Type":"ContainerDied","Data":"28c533f66db08e9ab2d83b939e44473d02ade549ea9505ccf680efb47f920f7e"}
Oct 30 20:40:03.870396 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:03.870378    5958 scope.go:122] "RemoveContainer" containerID="fccfc90067e5670504d2863baa8e9f90b7a954b37eb4a97d14c910198ef45f2d"
Oct 30 20:40:03.882797 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:03.882782    5958 scope.go:122] "RemoveContainer" containerID="fccfc90067e5670504d2863baa8e9f90b7a954b37eb4a97d14c910198ef45f2d"
Oct 30 20:40:03.883030 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:40:03.882999    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fccfc90067e5670504d2863baa8e9f90b7a954b37eb4a97d14c910198ef45f2d\": container with ID starting with fccfc90067e5670504d2863baa8e9f90b7a954b37eb4a97d14c910198ef45f2d not found: ID does not exist" containerID="fccfc90067e5670504d2863baa8e9f90b7a954b37eb4a97d14c910198ef45f2d"
Oct 30 20:40:03.883077 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:03.883041    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fccfc90067e5670504d2863baa8e9f90b7a954b37eb4a97d14c910198ef45f2d"} err="failed to get container status \"fccfc90067e5670504d2863baa8e9f90b7a954b37eb4a97d14c910198ef45f2d\": rpc error: code = NotFound desc = could not find container \"fccfc90067e5670504d2863baa8e9f90b7a954b37eb4a97d14c910198ef45f2d\": container with ID starting with fccfc90067e5670504d2863baa8e9f90b7a954b37eb4a97d14c910198ef45f2d not found: ID does not exist"
Oct 30 20:40:04.036829 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:04.036805    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj"]
Oct 30 20:40:04.086585 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:04.086555    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b2cfa0-747bc664b9-h7xhj"]
Oct 30 20:40:05.397202 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:05.397171    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e41a65a-4901-48f6-b3af-78dfe213c601" path="/var/lib/kubelet/pods/2e41a65a-4901-48f6-b3af-78dfe213c601/volumes"
Oct 30 20:40:05.854254 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:05.854218    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv" podUID="7419f2cd-8447-48ed-9005-2014f48336a1" containerName="payload-pre-processing" containerID="cri-o://1ff4d95bfdd02dd5b8af05c20b2952751b6db13c16f68e606087d7963784b631" gracePeriod=25
Oct 30 20:40:06.606353 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:40:06.606320    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:40:13.418962 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:13.418923    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv" podUID="7419f2cd-8447-48ed-9005-2014f48336a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:40:18.348211 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:18.348169    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"]
Oct 30 20:40:18.348747 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:18.348372    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm" podUID="4b333509-55f1-4d62-8bdc-154d2b47cc4b" containerName="istio-proxy" containerID="cri-o://9f7bb05b0773df1f5a93e2a159d2e1289809c329b64b269116bf39755209948f" gracePeriod=30
Oct 30 20:40:20.042731 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.042698    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sghxc"]
Oct 30 20:40:20.090671 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.090632    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sghxc"]
Oct 30 20:40:20.098117 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.098076    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-84mb7"]
Oct 30 20:40:20.098278 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.098129    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-84mb7"]
Oct 30 20:40:20.177117 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.177089    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-chfp4"]
Oct 30 20:40:20.177569 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.177555    5958 cpu_manager.go:414] "containerMap: removing container" podUID="2e41a65a-4901-48f6-b3af-78dfe213c601" containerName="maas-api"
Oct 30 20:40:20.177621 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.177571    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e41a65a-4901-48f6-b3af-78dfe213c601" containerName="maas-api"
Oct 30 20:40:20.177660 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.177646    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e41a65a-4901-48f6-b3af-78dfe213c601" containerName="maas-api"
Oct 30 20:40:20.180728 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.180712    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-chfp4"
Oct 30 20:40:20.234254 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.234226    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:40:20.234370 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.234258    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:40:20.234370 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.234308    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:40:20.234370 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.234329    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:40:20.240499 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.240475    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-chfp4"]
Oct 30 20:40:20.273027 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.273003    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtcf4\" (UniqueName: \"kubernetes.io/projected/88a0acb6-3ae8-44bf-bbf0-cf559290357e-kube-api-access-rtcf4\") pod \"maas-api-revoke-sub-3d4156f1-chfp4\" (UID: \"88a0acb6-3ae8-44bf-bbf0-cf559290357e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-chfp4"
Oct 30 20:40:20.273140 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.273042    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88a0acb6-3ae8-44bf-bbf0-cf559290357e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-chfp4\" (UID: \"88a0acb6-3ae8-44bf-bbf0-cf559290357e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-chfp4"
Oct 30 20:40:20.273140 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.273063    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88a0acb6-3ae8-44bf-bbf0-cf559290357e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-chfp4\" (UID: \"88a0acb6-3ae8-44bf-bbf0-cf559290357e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-chfp4"
Oct 30 20:40:20.373747 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.373717    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtcf4\" (UniqueName: \"kubernetes.io/projected/88a0acb6-3ae8-44bf-bbf0-cf559290357e-kube-api-access-rtcf4\") pod \"maas-api-revoke-sub-3d4156f1-chfp4\" (UID: \"88a0acb6-3ae8-44bf-bbf0-cf559290357e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-chfp4"
Oct 30 20:40:20.373890 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.373771    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88a0acb6-3ae8-44bf-bbf0-cf559290357e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-chfp4\" (UID: \"88a0acb6-3ae8-44bf-bbf0-cf559290357e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-chfp4"
Oct 30 20:40:20.373890 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.373797    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88a0acb6-3ae8-44bf-bbf0-cf559290357e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-chfp4\" (UID: \"88a0acb6-3ae8-44bf-bbf0-cf559290357e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-chfp4"
Oct 30 20:40:20.374497 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.374475    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88a0acb6-3ae8-44bf-bbf0-cf559290357e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-chfp4\" (UID: \"88a0acb6-3ae8-44bf-bbf0-cf559290357e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-chfp4"
Oct 30 20:40:20.376197 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.376174    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88a0acb6-3ae8-44bf-bbf0-cf559290357e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-chfp4\" (UID: \"88a0acb6-3ae8-44bf-bbf0-cf559290357e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-chfp4"
Oct 30 20:40:20.434487 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.434442    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtcf4\" (UniqueName: \"kubernetes.io/projected/88a0acb6-3ae8-44bf-bbf0-cf559290357e-kube-api-access-rtcf4\") pod \"maas-api-revoke-sub-3d4156f1-chfp4\" (UID: \"88a0acb6-3ae8-44bf-bbf0-cf559290357e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-chfp4"
Oct 30 20:40:20.492780 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.492755    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-chfp4"
Oct 30 20:40:20.867439 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.867411    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-chfp4"]
Oct 30 20:40:20.869467 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:40:20.869425    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a0acb6_3ae8_44bf_bbf0_cf559290357e.slice/crio-0e3eff02446c2f0ade333ca34ded2229c87cb383776785f3352e6f70c25d5742 WatchSource:0}: Error finding container 0e3eff02446c2f0ade333ca34ded2229c87cb383776785f3352e6f70c25d5742: Status 404 returned error can't find the container with id 0e3eff02446c2f0ade333ca34ded2229c87cb383776785f3352e6f70c25d5742
Oct 30 20:40:20.959320 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:20.959289    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-chfp4" event={"ID":"88a0acb6-3ae8-44bf-bbf0-cf559290357e","Type":"ContainerStarted","Data":"0e3eff02446c2f0ade333ca34ded2229c87cb383776785f3352e6f70c25d5742"}
Oct 30 20:40:21.397742 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:21.397678    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a55b702-8c17-4d5e-af8e-01be7863c500" path="/var/lib/kubelet/pods/0a55b702-8c17-4d5e-af8e-01be7863c500/volumes"
Oct 30 20:40:21.398074 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:21.398059    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a46cc1e5-5fa0-46e2-9cee-fbac39f676fb" path="/var/lib/kubelet/pods/a46cc1e5-5fa0-46e2-9cee-fbac39f676fb/volumes"
Oct 30 20:40:21.966040 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:21.966008    5958 generic.go:356] "Generic (PLEG): container finished" podID="88a0acb6-3ae8-44bf-bbf0-cf559290357e" containerID="e2205333cea1bf5eabc28a564d8dfc43d5f5dab740341030367dcd593e5eb61f" exitCode=0
Oct 30 20:40:21.966174 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:21.966080    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-chfp4" event={"ID":"88a0acb6-3ae8-44bf-bbf0-cf559290357e","Type":"ContainerDied","Data":"e2205333cea1bf5eabc28a564d8dfc43d5f5dab740341030367dcd593e5eb61f"}
Oct 30 20:40:21.970645 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:21.970620    5958 patch_prober.go:29] interesting pod/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:40:21.970760 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:21.970664    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm" podUID="4b333509-55f1-4d62-8bdc-154d2b47cc4b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:40:23.099656 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.099627    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-chfp4"
Oct 30 20:40:23.199840 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.199815    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88a0acb6-3ae8-44bf-bbf0-cf559290357e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88a0acb6-3ae8-44bf-bbf0-cf559290357e-maas-api-openshift-ca\") pod \"88a0acb6-3ae8-44bf-bbf0-cf559290357e\" (UID: \"88a0acb6-3ae8-44bf-bbf0-cf559290357e\") "
Oct 30 20:40:23.199972 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.199938    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88a0acb6-3ae8-44bf-bbf0-cf559290357e-kube-api-access-rtcf4\" (UniqueName: \"kubernetes.io/projected/88a0acb6-3ae8-44bf-bbf0-cf559290357e-kube-api-access-rtcf4\") pod \"88a0acb6-3ae8-44bf-bbf0-cf559290357e\" (UID: \"88a0acb6-3ae8-44bf-bbf0-cf559290357e\") "
Oct 30 20:40:23.200036 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.200019    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88a0acb6-3ae8-44bf-bbf0-cf559290357e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88a0acb6-3ae8-44bf-bbf0-cf559290357e-maas-api-xks-ca\") pod \"88a0acb6-3ae8-44bf-bbf0-cf559290357e\" (UID: \"88a0acb6-3ae8-44bf-bbf0-cf559290357e\") "
Oct 30 20:40:23.200137 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.200115    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a0acb6-3ae8-44bf-bbf0-cf559290357e-maas-api-openshift-ca" pod "88a0acb6-3ae8-44bf-bbf0-cf559290357e" (UID: "88a0acb6-3ae8-44bf-bbf0-cf559290357e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:40:23.200334 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.200319    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88a0acb6-3ae8-44bf-bbf0-cf559290357e-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:23.202148 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.202122    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a0acb6-3ae8-44bf-bbf0-cf559290357e-maas-api-xks-ca" pod "88a0acb6-3ae8-44bf-bbf0-cf559290357e" (UID: "88a0acb6-3ae8-44bf-bbf0-cf559290357e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:40:23.202233 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.202126    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a0acb6-3ae8-44bf-bbf0-cf559290357e-kube-api-access-rtcf4" pod "88a0acb6-3ae8-44bf-bbf0-cf559290357e" (UID: "88a0acb6-3ae8-44bf-bbf0-cf559290357e"). InnerVolumeSpecName "kube-api-access-rtcf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:40:23.300845 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.300783    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtcf4\" (UniqueName: \"kubernetes.io/projected/88a0acb6-3ae8-44bf-bbf0-cf559290357e-kube-api-access-rtcf4\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:23.300845 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.300806    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88a0acb6-3ae8-44bf-bbf0-cf559290357e-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:23.418971 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.418934    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv" podUID="7419f2cd-8447-48ed-9005-2014f48336a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:40:23.588416 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.588391    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:40:23.703355 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.703329    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-token\" (UniqueName: \"kubernetes.io/projected/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-token\") pod \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") "
Oct 30 20:40:23.703518 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.703393    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-envoy\") pod \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") "
Oct 30 20:40:23.703518 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.703424    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-workload-certs\") pod \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") "
Oct 30 20:40:23.703518 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.703487    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-credential-socket\") pod \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") "
Oct 30 20:40:23.703518 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.703505    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-workload-socket\") pod \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") "
Oct 30 20:40:23.703748 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.703529    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b333509-55f1-4d62-8bdc-154d2b47cc4b-kube-api-access-qsfmf\" (UniqueName: \"kubernetes.io/projected/4b333509-55f1-4d62-8bdc-154d2b47cc4b-kube-api-access-qsfmf\") pod \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") "
Oct 30 20:40:23.703748 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.703580    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-podinfo\") pod \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") "
Oct 30 20:40:23.703872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.703738    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istiod-ca-cert\") pod \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") "
Oct 30 20:40:23.703872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.703838    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-data\") pod \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\" (UID: \"4b333509-55f1-4d62-8bdc-154d2b47cc4b\") "
Oct 30 20:40:23.704183 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.703832    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-credential-socket" pod "4b333509-55f1-4d62-8bdc-154d2b47cc4b" (UID: "4b333509-55f1-4d62-8bdc-154d2b47cc4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:40:23.704183 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.703840    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-workload-socket" pod "4b333509-55f1-4d62-8bdc-154d2b47cc4b" (UID: "4b333509-55f1-4d62-8bdc-154d2b47cc4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:40:23.704375 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.704041    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-data" pod "4b333509-55f1-4d62-8bdc-154d2b47cc4b" (UID: "4b333509-55f1-4d62-8bdc-154d2b47cc4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:40:23.704375 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.704064    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-workload-certs" pod "4b333509-55f1-4d62-8bdc-154d2b47cc4b" (UID: "4b333509-55f1-4d62-8bdc-154d2b47cc4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:40:23.704375 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.704301    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istiod-ca-cert" pod "4b333509-55f1-4d62-8bdc-154d2b47cc4b" (UID: "4b333509-55f1-4d62-8bdc-154d2b47cc4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:40:23.704746 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.704431    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:23.704746 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.704469    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:23.704746 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.704484    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:23.704746 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.704498    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:23.704746 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.704512    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:23.705878 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.705853    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-token" pod "4b333509-55f1-4d62-8bdc-154d2b47cc4b" (UID: "4b333509-55f1-4d62-8bdc-154d2b47cc4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:40:23.705878 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.705857    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-envoy" pod "4b333509-55f1-4d62-8bdc-154d2b47cc4b" (UID: "4b333509-55f1-4d62-8bdc-154d2b47cc4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:40:23.706173 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.706154    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b333509-55f1-4d62-8bdc-154d2b47cc4b-kube-api-access-qsfmf" pod "4b333509-55f1-4d62-8bdc-154d2b47cc4b" (UID: "4b333509-55f1-4d62-8bdc-154d2b47cc4b"). InnerVolumeSpecName "kube-api-access-qsfmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:40:23.706352 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.706338    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-podinfo" pod "4b333509-55f1-4d62-8bdc-154d2b47cc4b" (UID: "4b333509-55f1-4d62-8bdc-154d2b47cc4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:40:23.805218 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.805184    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsfmf\" (UniqueName: \"kubernetes.io/projected/4b333509-55f1-4d62-8bdc-154d2b47cc4b-kube-api-access-qsfmf\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:23.805218 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.805209    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:23.805218 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.805219    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:23.805408 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.805228    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b333509-55f1-4d62-8bdc-154d2b47cc4b-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:23.979953 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.979888    5958 generic.go:356] "Generic (PLEG): container finished" podID="4b333509-55f1-4d62-8bdc-154d2b47cc4b" containerID="9f7bb05b0773df1f5a93e2a159d2e1289809c329b64b269116bf39755209948f" exitCode=0
Oct 30 20:40:23.980071 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.979957    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"
Oct 30 20:40:23.980071 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.979974    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm" event={"ID":"4b333509-55f1-4d62-8bdc-154d2b47cc4b","Type":"ContainerDied","Data":"9f7bb05b0773df1f5a93e2a159d2e1289809c329b64b269116bf39755209948f"}
Oct 30 20:40:23.980071 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.980030    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm" event={"ID":"4b333509-55f1-4d62-8bdc-154d2b47cc4b","Type":"ContainerDied","Data":"1e94b72174decb8051c718424dff4200927719b358e2d3f5eeaa94fcb3db89a6"}
Oct 30 20:40:23.980071 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.980054    5958 scope.go:122] "RemoveContainer" containerID="9f7bb05b0773df1f5a93e2a159d2e1289809c329b64b269116bf39755209948f"
Oct 30 20:40:23.981559 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.981538    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-chfp4" event={"ID":"88a0acb6-3ae8-44bf-bbf0-cf559290357e","Type":"ContainerDied","Data":"0e3eff02446c2f0ade333ca34ded2229c87cb383776785f3352e6f70c25d5742"}
Oct 30 20:40:23.981628 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.981580    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e3eff02446c2f0ade333ca34ded2229c87cb383776785f3352e6f70c25d5742"
Oct 30 20:40:23.981628 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.981579    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-chfp4"
Oct 30 20:40:23.990609 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.990592    5958 scope.go:122] "RemoveContainer" containerID="fd60a6f19f173dd2c48d40e42114a1107b138893b0093cdf453ce31ba1165f6c"
Oct 30 20:40:23.999190 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.999169    5958 scope.go:122] "RemoveContainer" containerID="9f7bb05b0773df1f5a93e2a159d2e1289809c329b64b269116bf39755209948f"
Oct 30 20:40:23.999484 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:40:23.999445    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f7bb05b0773df1f5a93e2a159d2e1289809c329b64b269116bf39755209948f\": container with ID starting with 9f7bb05b0773df1f5a93e2a159d2e1289809c329b64b269116bf39755209948f not found: ID does not exist" containerID="9f7bb05b0773df1f5a93e2a159d2e1289809c329b64b269116bf39755209948f"
Oct 30 20:40:23.999538 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.999492    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f7bb05b0773df1f5a93e2a159d2e1289809c329b64b269116bf39755209948f"} err="failed to get container status \"9f7bb05b0773df1f5a93e2a159d2e1289809c329b64b269116bf39755209948f\": rpc error: code = NotFound desc = could not find container \"9f7bb05b0773df1f5a93e2a159d2e1289809c329b64b269116bf39755209948f\": container with ID starting with 9f7bb05b0773df1f5a93e2a159d2e1289809c329b64b269116bf39755209948f not found: ID does not exist"
Oct 30 20:40:23.999538 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.999508    5958 scope.go:122] "RemoveContainer" containerID="fd60a6f19f173dd2c48d40e42114a1107b138893b0093cdf453ce31ba1165f6c"
Oct 30 20:40:23.999743 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:40:23.999727    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd60a6f19f173dd2c48d40e42114a1107b138893b0093cdf453ce31ba1165f6c\": container with ID starting with fd60a6f19f173dd2c48d40e42114a1107b138893b0093cdf453ce31ba1165f6c not found: ID does not exist" containerID="fd60a6f19f173dd2c48d40e42114a1107b138893b0093cdf453ce31ba1165f6c"
Oct 30 20:40:23.999790 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:23.999748    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd60a6f19f173dd2c48d40e42114a1107b138893b0093cdf453ce31ba1165f6c"} err="failed to get container status \"fd60a6f19f173dd2c48d40e42114a1107b138893b0093cdf453ce31ba1165f6c\": rpc error: code = NotFound desc = could not find container \"fd60a6f19f173dd2c48d40e42114a1107b138893b0093cdf453ce31ba1165f6c\": container with ID starting with fd60a6f19f173dd2c48d40e42114a1107b138893b0093cdf453ce31ba1165f6c not found: ID does not exist"
Oct 30 20:40:24.199480 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:24.199440    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"]
Oct 30 20:40:24.250007 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:24.249952    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b2cfa0-openshift-default-65cbd4964fjzzrm"]
Oct 30 20:40:25.397407 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:25.397365    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b333509-55f1-4d62-8bdc-154d2b47cc4b" path="/var/lib/kubelet/pods/4b333509-55f1-4d62-8bdc-154d2b47cc4b/volumes"
Oct 30 20:40:26.003197 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:26.003166    5958 generic.go:356] "Generic (PLEG): container finished" podID="7419f2cd-8447-48ed-9005-2014f48336a1" containerID="1ff4d95bfdd02dd5b8af05c20b2952751b6db13c16f68e606087d7963784b631" exitCode=0
Oct 30 20:40:26.003317 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:26.003228    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv" event={"ID":"7419f2cd-8447-48ed-9005-2014f48336a1","Type":"ContainerDied","Data":"1ff4d95bfdd02dd5b8af05c20b2952751b6db13c16f68e606087d7963784b631"}
Oct 30 20:40:26.101466 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:26.101431    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv"
Oct 30 20:40:26.224645 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:26.224538    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7419f2cd-8447-48ed-9005-2014f48336a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7419f2cd-8447-48ed-9005-2014f48336a1-plugins-config-volume\") pod \"7419f2cd-8447-48ed-9005-2014f48336a1\" (UID: \"7419f2cd-8447-48ed-9005-2014f48336a1\") "
Oct 30 20:40:26.224950 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:26.224926    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7419f2cd-8447-48ed-9005-2014f48336a1-plugins-config-volume" pod "7419f2cd-8447-48ed-9005-2014f48336a1" (UID: "7419f2cd-8447-48ed-9005-2014f48336a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:40:26.225120 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:26.224990    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7419f2cd-8447-48ed-9005-2014f48336a1-kube-api-access-b47z7\" (UniqueName: \"kubernetes.io/projected/7419f2cd-8447-48ed-9005-2014f48336a1-kube-api-access-b47z7\") pod \"7419f2cd-8447-48ed-9005-2014f48336a1\" (UID: \"7419f2cd-8447-48ed-9005-2014f48336a1\") "
Oct 30 20:40:26.225601 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:26.225577    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7419f2cd-8447-48ed-9005-2014f48336a1-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:26.227303 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:26.227276    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7419f2cd-8447-48ed-9005-2014f48336a1-kube-api-access-b47z7" pod "7419f2cd-8447-48ed-9005-2014f48336a1" (UID: "7419f2cd-8447-48ed-9005-2014f48336a1"). InnerVolumeSpecName "kube-api-access-b47z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:40:26.326847 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:26.326825    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b47z7\" (UniqueName: \"kubernetes.io/projected/7419f2cd-8447-48ed-9005-2014f48336a1-kube-api-access-b47z7\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:40:27.010347 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:27.010287    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv"
Oct 30 20:40:27.010738 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:27.010342    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv" event={"ID":"7419f2cd-8447-48ed-9005-2014f48336a1","Type":"ContainerDied","Data":"99950da8f8a5813b7fa4016d9d637db3b7d78c68f60ff85163189f66b6588240"}
Oct 30 20:40:27.010738 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:27.010402    5958 scope.go:122] "RemoveContainer" containerID="1ff4d95bfdd02dd5b8af05c20b2952751b6db13c16f68e606087d7963784b631"
Oct 30 20:40:27.177835 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:27.177801    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv"]
Oct 30 20:40:27.227683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:27.227659    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b2cfa0-7688dvtvrv"]
Oct 30 20:40:27.397045 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:27.397017    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7419f2cd-8447-48ed-9005-2014f48336a1" path="/var/lib/kubelet/pods/7419f2cd-8447-48ed-9005-2014f48336a1/volumes"
Oct 30 20:40:36.604153 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:40:36.604116    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:40:42.983976 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:42.983940    5958 scope.go:122] "RemoveContainer" containerID="44fa94d7a832bf97f8cf5a399de7e53b4394a915788cb758d57a0e205bdc7c6c"
Oct 30 20:40:42.992500 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:42.992478    5958 scope.go:122] "RemoveContainer" containerID="56dbcdd354e9c344650fa354af71fd4f573d6d48aa0c68efb3023711eebcca57"
Oct 30 20:40:43.000486 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:43.000465    5958 scope.go:122] "RemoveContainer" containerID="53981e61950139432f7ee784ddd3b071a557d63095d4fba10daab5914ad2d0b3"
Oct 30 20:40:51.053544 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:51.053510    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-f2hzt"]
Oct 30 20:40:51.103435 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:51.103410    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-f2hzt"]
Oct 30 20:40:51.396988 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:40:51.396954    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="242fe0e8-092a-4b34-8d79-e6f111215826" path="/var/lib/kubelet/pods/242fe0e8-092a-4b34-8d79-e6f111215826/volumes"
Oct 30 20:41:06.604936 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:41:06.604901    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:41:21.049604 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:41:21.049522    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qtghc"]
Oct 30 20:41:21.099398 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:41:21.099369    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-qtghc"]
Oct 30 20:41:21.396668 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:41:21.396637    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bbc4e41-d6cc-4a9e-8615-42924552ce70" path="/var/lib/kubelet/pods/1bbc4e41-d6cc-4a9e-8615-42924552ce70/volumes"
Oct 30 20:41:36.604563 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:41:36.604527    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:41:43.037712 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:41:43.037681    5958 scope.go:122] "RemoveContainer" containerID="3b0c764670e6fa38e21238377b9639a692d0fb6d8cdbcd64876e2b2adea24c0e"
Oct 30 20:41:45.050749 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:41:45.050714    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-llr8j"]
Oct 30 20:41:45.100203 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:41:45.100176    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-llr8j"]
Oct 30 20:41:45.396829 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:41:45.396801    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e" path="/var/lib/kubelet/pods/3a68fcf8-61a4-4e04-8ef4-43b13ba10c9e/volumes"
Oct 30 20:42:06.627920 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:42:06.627882    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:42:08.465132 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.465095    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nksc8"]
Oct 30 20:42:08.465741 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.465720    5958 cpu_manager.go:414] "containerMap: removing container" podUID="4b333509-55f1-4d62-8bdc-154d2b47cc4b" containerName="istio-proxy"
Oct 30 20:42:08.465836 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.465743    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b333509-55f1-4d62-8bdc-154d2b47cc4b" containerName="istio-proxy"
Oct 30 20:42:08.465836 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.465762    5958 cpu_manager.go:414] "containerMap: removing container" podUID="4b333509-55f1-4d62-8bdc-154d2b47cc4b" containerName="istio-proxy"
Oct 30 20:42:08.465836 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.465770    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b333509-55f1-4d62-8bdc-154d2b47cc4b" containerName="istio-proxy"
Oct 30 20:42:08.465836 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.465784    5958 cpu_manager.go:414] "containerMap: removing container" podUID="7419f2cd-8447-48ed-9005-2014f48336a1" containerName="payload-pre-processing"
Oct 30 20:42:08.465836 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.465792    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7419f2cd-8447-48ed-9005-2014f48336a1" containerName="payload-pre-processing"
Oct 30 20:42:08.465836 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.465804    5958 cpu_manager.go:414] "containerMap: removing container" podUID="88a0acb6-3ae8-44bf-bbf0-cf559290357e" containerName="revoke-keys"
Oct 30 20:42:08.465836 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.465811    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88a0acb6-3ae8-44bf-bbf0-cf559290357e" containerName="revoke-keys"
Oct 30 20:42:08.466186 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.465933    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b333509-55f1-4d62-8bdc-154d2b47cc4b" containerName="istio-proxy"
Oct 30 20:42:08.466186 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.465947    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b333509-55f1-4d62-8bdc-154d2b47cc4b" containerName="istio-proxy"
Oct 30 20:42:08.466186 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.465956    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="7419f2cd-8447-48ed-9005-2014f48336a1" containerName="payload-pre-processing"
Oct 30 20:42:08.466186 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.465965    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="88a0acb6-3ae8-44bf-bbf0-cf559290357e" containerName="revoke-keys"
Oct 30 20:42:08.470625 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.470602    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nksc8"
Oct 30 20:42:08.523393 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.523360    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:42:08.523555 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.523438    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:42:08.573065 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.573041    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:42:08.573179 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.573100    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:42:08.574601 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.574583    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nksc8"]
Oct 30 20:42:08.623362 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.623340    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c705b4-1d88-4a02-b59f-8f7bb9af6d43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-nksc8\" (UID: \"d5c705b4-1d88-4a02-b59f-8f7bb9af6d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nksc8"
Oct 30 20:42:08.623515 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.623377    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c705b4-1d88-4a02-b59f-8f7bb9af6d43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-nksc8\" (UID: \"d5c705b4-1d88-4a02-b59f-8f7bb9af6d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nksc8"
Oct 30 20:42:08.623515 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.623418    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgt6f\" (UniqueName: \"kubernetes.io/projected/d5c705b4-1d88-4a02-b59f-8f7bb9af6d43-kube-api-access-rgt6f\") pod \"maas-api-revoke-sub-d576e68b-nksc8\" (UID: \"d5c705b4-1d88-4a02-b59f-8f7bb9af6d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nksc8"
Oct 30 20:42:08.724657 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.724603    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c705b4-1d88-4a02-b59f-8f7bb9af6d43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-nksc8\" (UID: \"d5c705b4-1d88-4a02-b59f-8f7bb9af6d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nksc8"
Oct 30 20:42:08.724657 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.724639    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c705b4-1d88-4a02-b59f-8f7bb9af6d43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-nksc8\" (UID: \"d5c705b4-1d88-4a02-b59f-8f7bb9af6d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nksc8"
Oct 30 20:42:08.724785 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.724688    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgt6f\" (UniqueName: \"kubernetes.io/projected/d5c705b4-1d88-4a02-b59f-8f7bb9af6d43-kube-api-access-rgt6f\") pod \"maas-api-revoke-sub-d576e68b-nksc8\" (UID: \"d5c705b4-1d88-4a02-b59f-8f7bb9af6d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nksc8"
Oct 30 20:42:08.725329 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.725306    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c705b4-1d88-4a02-b59f-8f7bb9af6d43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-nksc8\" (UID: \"d5c705b4-1d88-4a02-b59f-8f7bb9af6d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nksc8"
Oct 30 20:42:08.727024 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.727004    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c705b4-1d88-4a02-b59f-8f7bb9af6d43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-nksc8\" (UID: \"d5c705b4-1d88-4a02-b59f-8f7bb9af6d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nksc8"
Oct 30 20:42:08.786230 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:08.786210    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgt6f\" (UniqueName: \"kubernetes.io/projected/d5c705b4-1d88-4a02-b59f-8f7bb9af6d43-kube-api-access-rgt6f\") pod \"maas-api-revoke-sub-d576e68b-nksc8\" (UID: \"d5c705b4-1d88-4a02-b59f-8f7bb9af6d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nksc8"
Oct 30 20:42:09.083340 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:09.083301    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nksc8"
Oct 30 20:42:09.457421 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:09.457399    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nksc8"]
Oct 30 20:42:09.459887 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:42:09.459862    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c705b4_1d88_4a02_b59f_8f7bb9af6d43.slice/crio-a063db794e2e0aeab905b9ff7792bf81e511193d9f4a6a8e364026880fdf1e8c WatchSource:0}: Error finding container a063db794e2e0aeab905b9ff7792bf81e511193d9f4a6a8e364026880fdf1e8c: Status 404 returned error can't find the container with id a063db794e2e0aeab905b9ff7792bf81e511193d9f4a6a8e364026880fdf1e8c
Oct 30 20:42:09.532894 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:09.532869    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nksc8" event={"ID":"d5c705b4-1d88-4a02-b59f-8f7bb9af6d43","Type":"ContainerStarted","Data":"a063db794e2e0aeab905b9ff7792bf81e511193d9f4a6a8e364026880fdf1e8c"}
Oct 30 20:42:10.539634 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:10.539601    5958 generic.go:356] "Generic (PLEG): container finished" podID="d5c705b4-1d88-4a02-b59f-8f7bb9af6d43" containerID="0be0d3a86bb61940e42db38f40d6764e2c3b4dd551c19887283e096b4266f190" exitCode=0
Oct 30 20:42:10.540000 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:10.539650    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nksc8" event={"ID":"d5c705b4-1d88-4a02-b59f-8f7bb9af6d43","Type":"ContainerDied","Data":"0be0d3a86bb61940e42db38f40d6764e2c3b4dd551c19887283e096b4266f190"}
Oct 30 20:42:11.675406 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:11.675384    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nksc8"
Oct 30 20:42:11.750424 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:11.750398    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c705b4-1d88-4a02-b59f-8f7bb9af6d43-kube-api-access-rgt6f\" (UniqueName: \"kubernetes.io/projected/d5c705b4-1d88-4a02-b59f-8f7bb9af6d43-kube-api-access-rgt6f\") pod \"d5c705b4-1d88-4a02-b59f-8f7bb9af6d43\" (UID: \"d5c705b4-1d88-4a02-b59f-8f7bb9af6d43\") "
Oct 30 20:42:11.750596 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:11.750506    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5c705b4-1d88-4a02-b59f-8f7bb9af6d43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c705b4-1d88-4a02-b59f-8f7bb9af6d43-maas-api-xks-ca\") pod \"d5c705b4-1d88-4a02-b59f-8f7bb9af6d43\" (UID: \"d5c705b4-1d88-4a02-b59f-8f7bb9af6d43\") "
Oct 30 20:42:11.750668 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:11.750616    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c705b4-1d88-4a02-b59f-8f7bb9af6d43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c705b4-1d88-4a02-b59f-8f7bb9af6d43-maas-api-openshift-ca\") pod \"d5c705b4-1d88-4a02-b59f-8f7bb9af6d43\" (UID: \"d5c705b4-1d88-4a02-b59f-8f7bb9af6d43\") "
Oct 30 20:42:11.750938 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:11.750918    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c705b4-1d88-4a02-b59f-8f7bb9af6d43-maas-api-openshift-ca" pod "d5c705b4-1d88-4a02-b59f-8f7bb9af6d43" (UID: "d5c705b4-1d88-4a02-b59f-8f7bb9af6d43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:42:11.752551 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:11.752523    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c705b4-1d88-4a02-b59f-8f7bb9af6d43-maas-api-xks-ca" pod "d5c705b4-1d88-4a02-b59f-8f7bb9af6d43" (UID: "d5c705b4-1d88-4a02-b59f-8f7bb9af6d43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:42:11.752687 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:11.752662    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c705b4-1d88-4a02-b59f-8f7bb9af6d43-kube-api-access-rgt6f" pod "d5c705b4-1d88-4a02-b59f-8f7bb9af6d43" (UID: "d5c705b4-1d88-4a02-b59f-8f7bb9af6d43"). InnerVolumeSpecName "kube-api-access-rgt6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:11.851795 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:11.851773    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c705b4-1d88-4a02-b59f-8f7bb9af6d43-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:11.851795 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:11.851795    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgt6f\" (UniqueName: \"kubernetes.io/projected/d5c705b4-1d88-4a02-b59f-8f7bb9af6d43-kube-api-access-rgt6f\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:11.851949 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:11.851805    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c705b4-1d88-4a02-b59f-8f7bb9af6d43-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:12.550600 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:12.550569    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nksc8" event={"ID":"d5c705b4-1d88-4a02-b59f-8f7bb9af6d43","Type":"ContainerDied","Data":"a063db794e2e0aeab905b9ff7792bf81e511193d9f4a6a8e364026880fdf1e8c"}
Oct 30 20:42:12.550600 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:12.550595    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nksc8"
Oct 30 20:42:12.550799 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:12.550615    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a063db794e2e0aeab905b9ff7792bf81e511193d9f4a6a8e364026880fdf1e8c"
Oct 30 20:42:30.163692 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:30.163654    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zmvlw"]
Oct 30 20:42:30.164283 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:30.164262    5958 cpu_manager.go:414] "containerMap: removing container" podUID="d5c705b4-1d88-4a02-b59f-8f7bb9af6d43" containerName="revoke-keys"
Oct 30 20:42:30.164377 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:30.164285    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c705b4-1d88-4a02-b59f-8f7bb9af6d43" containerName="revoke-keys"
Oct 30 20:42:30.164442 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:30.164432    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5c705b4-1d88-4a02-b59f-8f7bb9af6d43" containerName="revoke-keys"
Oct 30 20:42:30.167646 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:30.167624    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zmvlw"
Oct 30 20:42:30.235811 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:30.235774    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c224149-348e-48aa-ad3f-be2310c6f162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zmvlw\" (UID: \"1c224149-348e-48aa-ad3f-be2310c6f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zmvlw"
Oct 30 20:42:30.235986 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:30.235893    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c224149-348e-48aa-ad3f-be2310c6f162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zmvlw\" (UID: \"1c224149-348e-48aa-ad3f-be2310c6f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zmvlw"
Oct 30 20:42:30.235986 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:30.235968    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rblwk\" (UniqueName: \"kubernetes.io/projected/1c224149-348e-48aa-ad3f-be2310c6f162-kube-api-access-rblwk\") pod \"maas-api-revoke-sub-d2bc93bc-zmvlw\" (UID: \"1c224149-348e-48aa-ad3f-be2310c6f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zmvlw"
Oct 30 20:42:30.272918 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:30.272890    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:42:30.273074 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:30.273055    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:42:30.273153 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:30.273109    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:42:30.273221 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:30.273206    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:42:30.274586 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:30.274566    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zmvlw"]
Oct 30 20:42:30.337383 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:30.337359    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rblwk\" (UniqueName: \"kubernetes.io/projected/1c224149-348e-48aa-ad3f-be2310c6f162-kube-api-access-rblwk\") pod \"maas-api-revoke-sub-d2bc93bc-zmvlw\" (UID: \"1c224149-348e-48aa-ad3f-be2310c6f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zmvlw"
Oct 30 20:42:30.337532 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:30.337420    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c224149-348e-48aa-ad3f-be2310c6f162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zmvlw\" (UID: \"1c224149-348e-48aa-ad3f-be2310c6f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zmvlw"
Oct 30 20:42:30.337532 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:30.337480    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c224149-348e-48aa-ad3f-be2310c6f162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zmvlw\" (UID: \"1c224149-348e-48aa-ad3f-be2310c6f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zmvlw"
Oct 30 20:42:30.338050 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:30.338031    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c224149-348e-48aa-ad3f-be2310c6f162-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zmvlw\" (UID: \"1c224149-348e-48aa-ad3f-be2310c6f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zmvlw"
Oct 30 20:42:30.339756 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:30.339740    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c224149-348e-48aa-ad3f-be2310c6f162-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-zmvlw\" (UID: \"1c224149-348e-48aa-ad3f-be2310c6f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zmvlw"
Oct 30 20:42:30.399372 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:30.399344    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rblwk\" (UniqueName: \"kubernetes.io/projected/1c224149-348e-48aa-ad3f-be2310c6f162-kube-api-access-rblwk\") pod \"maas-api-revoke-sub-d2bc93bc-zmvlw\" (UID: \"1c224149-348e-48aa-ad3f-be2310c6f162\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zmvlw"
Oct 30 20:42:30.480892 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:30.480846    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zmvlw"
Oct 30 20:42:30.851700 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:30.851677    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zmvlw"]
Oct 30 20:42:30.853782 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:42:30.853744    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c224149_348e_48aa_ad3f_be2310c6f162.slice/crio-b9b757a23b4e8559b3b310f9254327d82dfef650a790e3f542d6a4e54d05ef6b WatchSource:0}: Error finding container b9b757a23b4e8559b3b310f9254327d82dfef650a790e3f542d6a4e54d05ef6b: Status 404 returned error can't find the container with id b9b757a23b4e8559b3b310f9254327d82dfef650a790e3f542d6a4e54d05ef6b
Oct 30 20:42:31.648030 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:31.648003    5958 generic.go:356] "Generic (PLEG): container finished" podID="1c224149-348e-48aa-ad3f-be2310c6f162" containerID="87da1707a7c71d20e10db578c478522ed82daa56f7e2207637ffa0b53a5cccda" exitCode=0
Oct 30 20:42:31.648324 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:31.648087    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zmvlw" event={"ID":"1c224149-348e-48aa-ad3f-be2310c6f162","Type":"ContainerDied","Data":"87da1707a7c71d20e10db578c478522ed82daa56f7e2207637ffa0b53a5cccda"}
Oct 30 20:42:31.648324 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:31.648140    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zmvlw" event={"ID":"1c224149-348e-48aa-ad3f-be2310c6f162","Type":"ContainerStarted","Data":"b9b757a23b4e8559b3b310f9254327d82dfef650a790e3f542d6a4e54d05ef6b"}
Oct 30 20:42:32.780819 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:32.780796    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zmvlw"
Oct 30 20:42:32.861750 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:32.861720    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c224149-348e-48aa-ad3f-be2310c6f162-kube-api-access-rblwk\" (UniqueName: \"kubernetes.io/projected/1c224149-348e-48aa-ad3f-be2310c6f162-kube-api-access-rblwk\") pod \"1c224149-348e-48aa-ad3f-be2310c6f162\" (UID: \"1c224149-348e-48aa-ad3f-be2310c6f162\") "
Oct 30 20:42:32.861879 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:32.861855    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c224149-348e-48aa-ad3f-be2310c6f162-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c224149-348e-48aa-ad3f-be2310c6f162-maas-api-openshift-ca\") pod \"1c224149-348e-48aa-ad3f-be2310c6f162\" (UID: \"1c224149-348e-48aa-ad3f-be2310c6f162\") "
Oct 30 20:42:32.861879 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:32.861874    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c224149-348e-48aa-ad3f-be2310c6f162-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c224149-348e-48aa-ad3f-be2310c6f162-maas-api-xks-ca\") pod \"1c224149-348e-48aa-ad3f-be2310c6f162\" (UID: \"1c224149-348e-48aa-ad3f-be2310c6f162\") "
Oct 30 20:42:32.862203 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:32.862180    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c224149-348e-48aa-ad3f-be2310c6f162-maas-api-openshift-ca" pod "1c224149-348e-48aa-ad3f-be2310c6f162" (UID: "1c224149-348e-48aa-ad3f-be2310c6f162"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:42:32.863833 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:32.863813    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c224149-348e-48aa-ad3f-be2310c6f162-kube-api-access-rblwk" pod "1c224149-348e-48aa-ad3f-be2310c6f162" (UID: "1c224149-348e-48aa-ad3f-be2310c6f162"). InnerVolumeSpecName "kube-api-access-rblwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:32.863922 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:32.863906    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c224149-348e-48aa-ad3f-be2310c6f162-maas-api-xks-ca" pod "1c224149-348e-48aa-ad3f-be2310c6f162" (UID: "1c224149-348e-48aa-ad3f-be2310c6f162"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:42:32.962906 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:32.962840    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c224149-348e-48aa-ad3f-be2310c6f162-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:32.962906 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:32.962867    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c224149-348e-48aa-ad3f-be2310c6f162-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:32.962906 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:32.962878    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rblwk\" (UniqueName: \"kubernetes.io/projected/1c224149-348e-48aa-ad3f-be2310c6f162-kube-api-access-rblwk\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:42:33.666030 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:33.665926    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zmvlw" event={"ID":"1c224149-348e-48aa-ad3f-be2310c6f162","Type":"ContainerDied","Data":"b9b757a23b4e8559b3b310f9254327d82dfef650a790e3f542d6a4e54d05ef6b"}
Oct 30 20:42:33.666030 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:33.665951    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zmvlw"
Oct 30 20:42:33.666030 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:33.666019    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9b757a23b4e8559b3b310f9254327d82dfef650a790e3f542d6a4e54d05ef6b"
Oct 30 20:42:36.605032 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:42:36.604994    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:42:41.987263 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:41.987232    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 20:42:42.001044 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:42.001019    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 20:42:42.833390 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:42.833357    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"]
Oct 30 20:42:42.833908 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:42.833888    5958 cpu_manager.go:414] "containerMap: removing container" podUID="1c224149-348e-48aa-ad3f-be2310c6f162" containerName="revoke-keys"
Oct 30 20:42:42.834019 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:42.833911    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c224149-348e-48aa-ad3f-be2310c6f162" containerName="revoke-keys"
Oct 30 20:42:42.834083 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:42.834029    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c224149-348e-48aa-ad3f-be2310c6f162" containerName="revoke-keys"
Oct 30 20:42:42.837024 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:42.837004    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:42.889847 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:42.889806    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-c530ce-openshift-default-dockercfg-nxjzd\""
Oct 30 20:42:42.948305 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:42.948268    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-workload-certs\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:42.948511 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:42.948391    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-token\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:42.948511 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:42.948439    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-workload-socket\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:42.948511 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:42.948502    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-podinfo\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:42.948660 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:42.948589    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-credential-socket\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:42.948660 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:42.948607    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltkb9\" (UniqueName: \"kubernetes.io/projected/c7a3317e-2347-410b-a308-496d6ed9b2d7-kube-api-access-ltkb9\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:42.948660 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:42.948637    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-data\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:42.948767 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:42.948660    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-envoy\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:42.948767 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:42.948724    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a3317e-2347-410b-a308-496d6ed9b2d7-istiod-ca-cert\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:42.958629 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:42.958600    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"]
Oct 30 20:42:43.049289 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.049257    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-workload-socket\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:43.049763 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.049311    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-podinfo\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:43.049763 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.049430    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-credential-socket\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:43.049763 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.049489    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltkb9\" (UniqueName: \"kubernetes.io/projected/c7a3317e-2347-410b-a308-496d6ed9b2d7-kube-api-access-ltkb9\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:43.049763 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.049538    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-data\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:43.049763 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.049572    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-envoy\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:43.049763 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.049603    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a3317e-2347-410b-a308-496d6ed9b2d7-istiod-ca-cert\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:43.049763 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.049665    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-workload-certs\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:43.049763 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.049759    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-token\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:43.050497 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.049751    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-workload-socket\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:43.050497 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.049821    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-credential-socket\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:43.050497 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.050304    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-data\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:43.050619 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.050554    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-workload-certs\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:43.050867 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.050844    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a3317e-2347-410b-a308-496d6ed9b2d7-istiod-ca-cert\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:43.052327 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.052302    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-podinfo\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:43.053060 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.053038    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-envoy\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:43.063009 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.062983    5958 scope.go:122] "RemoveContainer" containerID="e3ee0d3c4c1eca4cbe7185311c6a1d19b2f4a9e1c9e9a748b89a16cc723f2dc4"
Oct 30 20:42:43.074596 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.074566    5958 scope.go:122] "RemoveContainer" containerID="0a70e687cbc4a77855a097489dfc4283b98f278917c1575411f2c4b4cb024ae9"
Oct 30 20:42:43.082174 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.082144    5958 scope.go:122] "RemoveContainer" containerID="cb5fa0186b8346240bc68087553c3609c3b4578d546d5d2ef5e869bf1bd6d18a"
Oct 30 20:42:43.110558 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.110538    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-token\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:43.113967 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.113946    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltkb9\" (UniqueName: \"kubernetes.io/projected/c7a3317e-2347-410b-a308-496d6ed9b2d7-kube-api-access-ltkb9\") pod \"e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") " pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:43.149827 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.149803    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:43.580418 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.580393    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"]
Oct 30 20:42:43.582517 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:42:43.582489    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a3317e_2347_410b_a308_496d6ed9b2d7.slice/crio-621be02490da4895abff6a9b84b80991a5ef8eaf532871f5e0d3794901fdedde WatchSource:0}: Error finding container 621be02490da4895abff6a9b84b80991a5ef8eaf532871f5e0d3794901fdedde: Status 404 returned error can't find the container with id 621be02490da4895abff6a9b84b80991a5ef8eaf532871f5e0d3794901fdedde
Oct 30 20:42:43.584615 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.584582    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:43.584694 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.584644    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:43.584694 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.584672    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:42:43.723502 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.723433    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq" event={"ID":"c7a3317e-2347-410b-a308-496d6ed9b2d7","Type":"ContainerStarted","Data":"d01a53dd998785d0c615068e1fa68ec12a034e9ad9a06f0e9352e68e65c850a0"}
Oct 30 20:42:43.723767 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.723748    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq" event={"ID":"c7a3317e-2347-410b-a308-496d6ed9b2d7","Type":"ContainerStarted","Data":"621be02490da4895abff6a9b84b80991a5ef8eaf532871f5e0d3794901fdedde"}
Oct 30 20:42:43.846341 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:43.846245    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq" podStartSLOduration=1.846231269 podStartE2EDuration="1.846231269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:43.84503842 +0000 UTC m=+3602.577253832" watchObservedRunningTime="2026-10-30 20:42:43.846231269 +0000 UTC m=+3602.578446658"
Oct 30 20:42:44.150267 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:44.150195    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:44.155033 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:44.155010    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:44.729531 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:44.729502    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:44.730406 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:44.730383    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:42:50.743975 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:50.743891    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c"]
Oct 30 20:42:50.748991 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:50.748961    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c"
Oct 30 20:42:50.806895 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:50.806865    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c"]
Oct 30 20:42:50.854156 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:50.854121    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-c530ce\""
Oct 30 20:42:50.854300 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:50.854223    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-304f993e-dockercfg-bpnj5\""
Oct 30 20:42:50.927247 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:50.927216    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbgj9\" (UniqueName: \"kubernetes.io/projected/4ebc5902-30bc-4d36-9514-398fa2345eb7-kube-api-access-xbgj9\") pod \"payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c\" (UID: \"4ebc5902-30bc-4d36-9514-398fa2345eb7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c"
Oct 30 20:42:50.927578 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:50.927549    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ebc5902-30bc-4d36-9514-398fa2345eb7-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c\" (UID: \"4ebc5902-30bc-4d36-9514-398fa2345eb7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c"
Oct 30 20:42:51.028864 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:51.028803    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ebc5902-30bc-4d36-9514-398fa2345eb7-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c\" (UID: \"4ebc5902-30bc-4d36-9514-398fa2345eb7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c"
Oct 30 20:42:51.028864 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:51.028862    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbgj9\" (UniqueName: \"kubernetes.io/projected/4ebc5902-30bc-4d36-9514-398fa2345eb7-kube-api-access-xbgj9\") pod \"payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c\" (UID: \"4ebc5902-30bc-4d36-9514-398fa2345eb7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c"
Oct 30 20:42:51.029464 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:51.029431    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ebc5902-30bc-4d36-9514-398fa2345eb7-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c\" (UID: \"4ebc5902-30bc-4d36-9514-398fa2345eb7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c"
Oct 30 20:42:51.090365 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:51.090340    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbgj9\" (UniqueName: \"kubernetes.io/projected/4ebc5902-30bc-4d36-9514-398fa2345eb7-kube-api-access-xbgj9\") pod \"payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c\" (UID: \"4ebc5902-30bc-4d36-9514-398fa2345eb7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c"
Oct 30 20:42:51.366001 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:51.365972    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c"
Oct 30 20:42:51.743218 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:51.743193    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c"]
Oct 30 20:42:51.745668 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:42:51.745641    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ebc5902_30bc_4d36_9514_398fa2345eb7.slice/crio-ff2d3b5e46cb0db0d59fc843f4e0be6e62b66a03abb8b1e438e812e77ae5e2b6 WatchSource:0}: Error finding container ff2d3b5e46cb0db0d59fc843f4e0be6e62b66a03abb8b1e438e812e77ae5e2b6: Status 404 returned error can't find the container with id ff2d3b5e46cb0db0d59fc843f4e0be6e62b66a03abb8b1e438e812e77ae5e2b6
Oct 30 20:42:51.766731 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:51.766705    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c" event={"ID":"4ebc5902-30bc-4d36-9514-398fa2345eb7","Type":"ContainerStarted","Data":"ff2d3b5e46cb0db0d59fc843f4e0be6e62b66a03abb8b1e438e812e77ae5e2b6"}
Oct 30 20:42:52.773582 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:52.773542    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c" event={"ID":"4ebc5902-30bc-4d36-9514-398fa2345eb7","Type":"ContainerStarted","Data":"c9d32062f7d84a927c585a2323bb354c72cad76b7694641ef9d3fd0708948c61"}
Oct 30 20:42:52.773934 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:52.773649    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c"
Oct 30 20:42:52.879186 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:52.879152    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx"]
Oct 30 20:42:52.882819 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:52.882800    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx"
Oct 30 20:42:52.935601 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:52.935565    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:42:52.941345 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:52.941302    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-c530ce-serving-cert\""
Oct 30 20:42:52.941523 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:52.941347    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:42:52.941523 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:52.941354    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xtx52\""
Oct 30 20:42:52.941523 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:52.941395    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 20:42:52.972981 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:52.972926    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c" podStartSLOduration=2.972909789 podStartE2EDuration="2.972909789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:52.915770896 +0000 UTC m=+3611.647986286" watchObservedRunningTime="2026-10-30 20:42:52.972909789 +0000 UTC m=+3611.705125183"
Oct 30 20:42:52.973653 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:52.973633    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx"]
Oct 30 20:42:53.052499 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:53.052393    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/327b42ff-e064-4adc-b764-dda3a7afde15-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx\" (UID: \"327b42ff-e064-4adc-b764-dda3a7afde15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx"
Oct 30 20:42:53.052666 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:53.052523    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/327b42ff-e064-4adc-b764-dda3a7afde15-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx\" (UID: \"327b42ff-e064-4adc-b764-dda3a7afde15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx"
Oct 30 20:42:53.052666 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:53.052591    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpcdk\" (UniqueName: \"kubernetes.io/projected/327b42ff-e064-4adc-b764-dda3a7afde15-kube-api-access-cpcdk\") pod \"maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx\" (UID: \"327b42ff-e064-4adc-b764-dda3a7afde15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx"
Oct 30 20:42:53.052666 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:53.052663    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/327b42ff-e064-4adc-b764-dda3a7afde15-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx\" (UID: \"327b42ff-e064-4adc-b764-dda3a7afde15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx"
Oct 30 20:42:53.154169 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:53.154127    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/327b42ff-e064-4adc-b764-dda3a7afde15-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx\" (UID: \"327b42ff-e064-4adc-b764-dda3a7afde15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx"
Oct 30 20:42:53.154359 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:53.154203    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpcdk\" (UniqueName: \"kubernetes.io/projected/327b42ff-e064-4adc-b764-dda3a7afde15-kube-api-access-cpcdk\") pod \"maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx\" (UID: \"327b42ff-e064-4adc-b764-dda3a7afde15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx"
Oct 30 20:42:53.154359 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:53.154238    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/327b42ff-e064-4adc-b764-dda3a7afde15-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx\" (UID: \"327b42ff-e064-4adc-b764-dda3a7afde15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx"
Oct 30 20:42:53.154359 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:53.154278    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/327b42ff-e064-4adc-b764-dda3a7afde15-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx\" (UID: \"327b42ff-e064-4adc-b764-dda3a7afde15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx"
Oct 30 20:42:53.154960 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:53.154925    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/327b42ff-e064-4adc-b764-dda3a7afde15-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx\" (UID: \"327b42ff-e064-4adc-b764-dda3a7afde15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx"
Oct 30 20:42:53.156737 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:53.156710    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/327b42ff-e064-4adc-b764-dda3a7afde15-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx\" (UID: \"327b42ff-e064-4adc-b764-dda3a7afde15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx"
Oct 30 20:42:53.156857 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:53.156833    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/327b42ff-e064-4adc-b764-dda3a7afde15-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx\" (UID: \"327b42ff-e064-4adc-b764-dda3a7afde15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx"
Oct 30 20:42:53.216169 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:53.216136    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpcdk\" (UniqueName: \"kubernetes.io/projected/327b42ff-e064-4adc-b764-dda3a7afde15-kube-api-access-cpcdk\") pod \"maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx\" (UID: \"327b42ff-e064-4adc-b764-dda3a7afde15\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx"
Oct 30 20:42:53.505734 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:53.505705    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx"
Oct 30 20:42:53.881552 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:53.881394    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx"]
Oct 30 20:42:53.883996 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:42:53.883963    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod327b42ff_e064_4adc_b764_dda3a7afde15.slice/crio-9e58745c8a4076b9a0714436c565c603c711342a25ce8a6995a78d9eb6a29a28 WatchSource:0}: Error finding container 9e58745c8a4076b9a0714436c565c603c711342a25ce8a6995a78d9eb6a29a28: Status 404 returned error can't find the container with id 9e58745c8a4076b9a0714436c565c603c711342a25ce8a6995a78d9eb6a29a28
Oct 30 20:42:54.783818 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:54.783786    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx" event={"ID":"327b42ff-e064-4adc-b764-dda3a7afde15","Type":"ContainerStarted","Data":"9e58745c8a4076b9a0714436c565c603c711342a25ce8a6995a78d9eb6a29a28"}
Oct 30 20:42:55.790657 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:55.790606    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx" event={"ID":"327b42ff-e064-4adc-b764-dda3a7afde15","Type":"ContainerStarted","Data":"b7e1ff68b368354a01d9fcd6ad258604e5c233ec9609121811e8586e94669151"}
Oct 30 20:42:55.791063 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:55.790678    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx"
Oct 30 20:42:55.909994 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:42:55.909946    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx" podStartSLOduration=2.845326737 podStartE2EDuration="3.909931271s" totalImagesPullingTime="1.064604534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:55.908159268 +0000 UTC m=+3614.640374657" watchObservedRunningTime="2026-10-30 20:42:55.909931271 +0000 UTC m=+3614.642146658"
Oct 30 20:43:01.801229 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:43:01.801195    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx"
Oct 30 20:43:03.781361 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:43:03.781330    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c"
Oct 30 20:43:06.615485 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:43:06.613769    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:43:09.893073 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:43:09.893043    5958 generic.go:356] "Generic (PLEG): container finished" podID="c7a3317e-2347-410b-a308-496d6ed9b2d7" containerID="d01a53dd998785d0c615068e1fa68ec12a034e9ad9a06f0e9352e68e65c850a0" exitCode=0
Oct 30 20:43:09.893525 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:43:09.893114    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq" event={"ID":"c7a3317e-2347-410b-a308-496d6ed9b2d7","Type":"ContainerDied","Data":"d01a53dd998785d0c615068e1fa68ec12a034e9ad9a06f0e9352e68e65c850a0"}
Oct 30 20:43:09.893525 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:43:09.893495    5958 scope.go:122] "RemoveContainer" containerID="d01a53dd998785d0c615068e1fa68ec12a034e9ad9a06f0e9352e68e65c850a0"
Oct 30 20:43:09.894599 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:43:09.894570    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:43:09.894695 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:43:09.894648    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:43:09.894695 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:43:09.894687    5958 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 20:43:10.899887 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:43:10.899849    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq" event={"ID":"c7a3317e-2347-410b-a308-496d6ed9b2d7","Type":"ContainerStarted","Data":"b64d85b9cf28380e9be1c2b2c4e1a0aece7e01b10a0113749246123f4b3e3cdb"}
Oct 30 20:43:11.158533 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:43:11.157821    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:43:11.160633 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:43:11.160600    5958 patch_prober.go:29] interesting pod/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 30 20:43:11.160783 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:43:11.160682    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq" podUID="c7a3317e-2347-410b-a308-496d6ed9b2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 30 20:43:12.157477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:43:12.156819    5958 patch_prober.go:29] interesting pod/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 30 20:43:12.157477 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:43:12.156908    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq" podUID="c7a3317e-2347-410b-a308-496d6ed9b2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 30 20:43:13.150848 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:43:13.150809    5958 patch_prober.go:29] interesting pod/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 30 20:43:13.151230 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:43:13.151200    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq" podUID="c7a3317e-2347-410b-a308-496d6ed9b2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 30 20:43:14.153876 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:43:14.153837    5958 patch_prober.go:29] interesting pod/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 30 20:43:14.154512 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:43:14.154479    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq" podUID="c7a3317e-2347-410b-a308-496d6ed9b2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 30 20:43:14.730503 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:43:14.730437    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:43:15.154484 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:43:15.153841    5958 patch_prober.go:29] interesting pod/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 30 20:43:15.154484 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:43:15.153913    5958 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq" podUID="c7a3317e-2347-410b-a308-496d6ed9b2d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 30 20:43:16.157949 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:43:16.157876    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:43:16.159513 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:43:16.159491    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:43:36.605918 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:43:36.605881    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:43:43.108229 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:43:43.108192    5958 scope.go:122] "RemoveContainer" containerID="c83af0b3c6e645378cdd0845cad0dd96248a94fe2a0e3f236752972d07a7bc03"
Oct 30 20:43:43.118600 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:43:43.118574    5958 scope.go:122] "RemoveContainer" containerID="e23d298448d7962766c350d327a08da5762bbab48b6addd0c90a05c5bb20ef90"
Oct 30 20:44:06.605202 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:44:06.605163    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:44:09.332380 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:09.332351    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c"]
Oct 30 20:44:10.514373 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:10.514343    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx"]
Oct 30 20:44:10.514755 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:10.514547    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx" podUID="327b42ff-e064-4adc-b764-dda3a7afde15" containerName="maas-api" containerID="cri-o://b7e1ff68b368354a01d9fcd6ad258604e5c233ec9609121811e8586e94669151" gracePeriod=30
Oct 30 20:44:11.762954 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:11.762925    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx"
Oct 30 20:44:11.928403 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:11.928319    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/327b42ff-e064-4adc-b764-dda3a7afde15-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/327b42ff-e064-4adc-b764-dda3a7afde15-maas-api-tls\") pod \"327b42ff-e064-4adc-b764-dda3a7afde15\" (UID: \"327b42ff-e064-4adc-b764-dda3a7afde15\") "
Oct 30 20:44:11.928403 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:11.928379    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/327b42ff-e064-4adc-b764-dda3a7afde15-metrics-certs\" (UniqueName: \"kubernetes.io/secret/327b42ff-e064-4adc-b764-dda3a7afde15-metrics-certs\") pod \"327b42ff-e064-4adc-b764-dda3a7afde15\" (UID: \"327b42ff-e064-4adc-b764-dda3a7afde15\") "
Oct 30 20:44:11.928664 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:11.928498    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/327b42ff-e064-4adc-b764-dda3a7afde15-kube-api-access-cpcdk\" (UniqueName: \"kubernetes.io/projected/327b42ff-e064-4adc-b764-dda3a7afde15-kube-api-access-cpcdk\") pod \"327b42ff-e064-4adc-b764-dda3a7afde15\" (UID: \"327b42ff-e064-4adc-b764-dda3a7afde15\") "
Oct 30 20:44:11.928664 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:11.928546    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/327b42ff-e064-4adc-b764-dda3a7afde15-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/327b42ff-e064-4adc-b764-dda3a7afde15-openshift-service-ca\") pod \"327b42ff-e064-4adc-b764-dda3a7afde15\" (UID: \"327b42ff-e064-4adc-b764-dda3a7afde15\") "
Oct 30 20:44:11.928965 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:11.928939    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/327b42ff-e064-4adc-b764-dda3a7afde15-openshift-service-ca" pod "327b42ff-e064-4adc-b764-dda3a7afde15" (UID: "327b42ff-e064-4adc-b764-dda3a7afde15"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:44:11.930765 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:11.930747    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/327b42ff-e064-4adc-b764-dda3a7afde15-kube-api-access-cpcdk" pod "327b42ff-e064-4adc-b764-dda3a7afde15" (UID: "327b42ff-e064-4adc-b764-dda3a7afde15"). InnerVolumeSpecName "kube-api-access-cpcdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:11.931030 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:11.931013    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/327b42ff-e064-4adc-b764-dda3a7afde15-metrics-certs" pod "327b42ff-e064-4adc-b764-dda3a7afde15" (UID: "327b42ff-e064-4adc-b764-dda3a7afde15"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:44:11.931075 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:11.931044    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/327b42ff-e064-4adc-b764-dda3a7afde15-maas-api-tls" pod "327b42ff-e064-4adc-b764-dda3a7afde15" (UID: "327b42ff-e064-4adc-b764-dda3a7afde15"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:44:12.029927 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:12.029896    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/327b42ff-e064-4adc-b764-dda3a7afde15-maas-api-tls\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:12.029927 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:12.029926    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/327b42ff-e064-4adc-b764-dda3a7afde15-metrics-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:12.030080 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:12.029941    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpcdk\" (UniqueName: \"kubernetes.io/projected/327b42ff-e064-4adc-b764-dda3a7afde15-kube-api-access-cpcdk\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:12.030080 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:12.029955    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/327b42ff-e064-4adc-b764-dda3a7afde15-openshift-service-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:12.204984 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:12.204908    5958 generic.go:356] "Generic (PLEG): container finished" podID="327b42ff-e064-4adc-b764-dda3a7afde15" containerID="b7e1ff68b368354a01d9fcd6ad258604e5c233ec9609121811e8586e94669151" exitCode=0
Oct 30 20:44:12.204984 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:12.204954    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx"
Oct 30 20:44:12.205164 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:12.204993    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx" event={"ID":"327b42ff-e064-4adc-b764-dda3a7afde15","Type":"ContainerDied","Data":"b7e1ff68b368354a01d9fcd6ad258604e5c233ec9609121811e8586e94669151"}
Oct 30 20:44:12.205164 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:12.205049    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx" event={"ID":"327b42ff-e064-4adc-b764-dda3a7afde15","Type":"ContainerDied","Data":"9e58745c8a4076b9a0714436c565c603c711342a25ce8a6995a78d9eb6a29a28"}
Oct 30 20:44:12.205164 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:12.205071    5958 scope.go:122] "RemoveContainer" containerID="b7e1ff68b368354a01d9fcd6ad258604e5c233ec9609121811e8586e94669151"
Oct 30 20:44:12.214230 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:12.214212    5958 scope.go:122] "RemoveContainer" containerID="b7e1ff68b368354a01d9fcd6ad258604e5c233ec9609121811e8586e94669151"
Oct 30 20:44:12.214509 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:44:12.214487    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7e1ff68b368354a01d9fcd6ad258604e5c233ec9609121811e8586e94669151\": container with ID starting with b7e1ff68b368354a01d9fcd6ad258604e5c233ec9609121811e8586e94669151 not found: ID does not exist" containerID="b7e1ff68b368354a01d9fcd6ad258604e5c233ec9609121811e8586e94669151"
Oct 30 20:44:12.214572 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:12.214517    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7e1ff68b368354a01d9fcd6ad258604e5c233ec9609121811e8586e94669151"} err="failed to get container status \"b7e1ff68b368354a01d9fcd6ad258604e5c233ec9609121811e8586e94669151\": rpc error: code = NotFound desc = could not find container \"b7e1ff68b368354a01d9fcd6ad258604e5c233ec9609121811e8586e94669151\": container with ID starting with b7e1ff68b368354a01d9fcd6ad258604e5c233ec9609121811e8586e94669151 not found: ID does not exist"
Oct 30 20:44:12.368987 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:12.368954    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx"]
Oct 30 20:44:12.418780 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:12.418755    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c530ce-ddfb6b89d-75vlx"]
Oct 30 20:44:13.396435 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:13.396396    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="327b42ff-e064-4adc-b764-dda3a7afde15" path="/var/lib/kubelet/pods/327b42ff-e064-4adc-b764-dda3a7afde15/volumes"
Oct 30 20:44:14.333114 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:14.333069    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c" podUID="4ebc5902-30bc-4d36-9514-398fa2345eb7" containerName="payload-processing" containerID="cri-o://c9d32062f7d84a927c585a2323bb354c72cad76b7694641ef9d3fd0708948c61" gracePeriod=25
Oct 30 20:44:22.539144 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:22.539057    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"]
Oct 30 20:44:22.539574 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:22.539287    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq" podUID="c7a3317e-2347-410b-a308-496d6ed9b2d7" containerName="istio-proxy" containerID="cri-o://b64d85b9cf28380e9be1c2b2c4e1a0aece7e01b10a0113749246123f4b3e3cdb" gracePeriod=30
Oct 30 20:44:23.779726 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:23.779685    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c" podUID="4ebc5902-30bc-4d36-9514-398fa2345eb7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:44:27.793098 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.793073    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:44:27.870615 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.870584    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7a3317e-2347-410b-a308-496d6ed9b2d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a3317e-2347-410b-a308-496d6ed9b2d7-istiod-ca-cert\") pod \"c7a3317e-2347-410b-a308-496d6ed9b2d7\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") "
Oct 30 20:44:27.870739 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.870635    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-credential-socket\") pod \"c7a3317e-2347-410b-a308-496d6ed9b2d7\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") "
Oct 30 20:44:27.870739 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.870687    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-token\" (UniqueName: \"kubernetes.io/projected/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-token\") pod \"c7a3317e-2347-410b-a308-496d6ed9b2d7\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") "
Oct 30 20:44:27.870739 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.870706    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-podinfo\") pod \"c7a3317e-2347-410b-a308-496d6ed9b2d7\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") "
Oct 30 20:44:27.870886 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.870740    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-workload-certs\") pod \"c7a3317e-2347-410b-a308-496d6ed9b2d7\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") "
Oct 30 20:44:27.870886 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.870800    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a3317e-2347-410b-a308-496d6ed9b2d7-kube-api-access-ltkb9\" (UniqueName: \"kubernetes.io/projected/c7a3317e-2347-410b-a308-496d6ed9b2d7-kube-api-access-ltkb9\") pod \"c7a3317e-2347-410b-a308-496d6ed9b2d7\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") "
Oct 30 20:44:27.870886 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.870825    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-data\") pod \"c7a3317e-2347-410b-a308-496d6ed9b2d7\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") "
Oct 30 20:44:27.870886 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.870867    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-envoy\") pod \"c7a3317e-2347-410b-a308-496d6ed9b2d7\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") "
Oct 30 20:44:27.871091 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.870939    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-workload-socket\") pod \"c7a3317e-2347-410b-a308-496d6ed9b2d7\" (UID: \"c7a3317e-2347-410b-a308-496d6ed9b2d7\") "
Oct 30 20:44:27.871091 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.870947    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-credential-socket" pod "c7a3317e-2347-410b-a308-496d6ed9b2d7" (UID: "c7a3317e-2347-410b-a308-496d6ed9b2d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:27.871091 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.870969    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a3317e-2347-410b-a308-496d6ed9b2d7-istiod-ca-cert" pod "c7a3317e-2347-410b-a308-496d6ed9b2d7" (UID: "c7a3317e-2347-410b-a308-496d6ed9b2d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:44:27.871259 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.871101    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-workload-certs" pod "c7a3317e-2347-410b-a308-496d6ed9b2d7" (UID: "c7a3317e-2347-410b-a308-496d6ed9b2d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:27.871259 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.871160    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-data" pod "c7a3317e-2347-410b-a308-496d6ed9b2d7" (UID: "c7a3317e-2347-410b-a308-496d6ed9b2d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:27.871363 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.871314    5958 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-credential-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:27.871363 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.871321    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-workload-socket" pod "c7a3317e-2347-410b-a308-496d6ed9b2d7" (UID: "c7a3317e-2347-410b-a308-496d6ed9b2d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:27.871363 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.871345    5958 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-workload-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:27.871363 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.871355    5958 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-data\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:27.871363 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.871364    5958 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a3317e-2347-410b-a308-496d6ed9b2d7-istiod-ca-cert\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:27.873011 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.872988    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-podinfo" pod "c7a3317e-2347-410b-a308-496d6ed9b2d7" (UID: "c7a3317e-2347-410b-a308-496d6ed9b2d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:44:27.873227 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.873203    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-envoy" pod "c7a3317e-2347-410b-a308-496d6ed9b2d7" (UID: "c7a3317e-2347-410b-a308-496d6ed9b2d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:44:27.873526 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.873511    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a3317e-2347-410b-a308-496d6ed9b2d7-kube-api-access-ltkb9" pod "c7a3317e-2347-410b-a308-496d6ed9b2d7" (UID: "c7a3317e-2347-410b-a308-496d6ed9b2d7"). InnerVolumeSpecName "kube-api-access-ltkb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:27.873588 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.873538    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-token" pod "c7a3317e-2347-410b-a308-496d6ed9b2d7" (UID: "c7a3317e-2347-410b-a308-496d6ed9b2d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:27.972210 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.972183    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltkb9\" (UniqueName: \"kubernetes.io/projected/c7a3317e-2347-410b-a308-496d6ed9b2d7-kube-api-access-ltkb9\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:27.972332 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.972211    5958 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-envoy\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:27.972332 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.972227    5958 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a3317e-2347-410b-a308-496d6ed9b2d7-workload-socket\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:27.972332 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.972238    5958 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-token\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:27.972332 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:27.972246    5958 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a3317e-2347-410b-a308-496d6ed9b2d7-istio-podinfo\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:28.290061 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:28.289984    5958 generic.go:356] "Generic (PLEG): container finished" podID="c7a3317e-2347-410b-a308-496d6ed9b2d7" containerID="b64d85b9cf28380e9be1c2b2c4e1a0aece7e01b10a0113749246123f4b3e3cdb" exitCode=0
Oct 30 20:44:28.290061 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:28.290031    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"
Oct 30 20:44:28.290234 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:28.290061    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq" event={"ID":"c7a3317e-2347-410b-a308-496d6ed9b2d7","Type":"ContainerDied","Data":"b64d85b9cf28380e9be1c2b2c4e1a0aece7e01b10a0113749246123f4b3e3cdb"}
Oct 30 20:44:28.290234 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:28.290116    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq" event={"ID":"c7a3317e-2347-410b-a308-496d6ed9b2d7","Type":"ContainerDied","Data":"621be02490da4895abff6a9b84b80991a5ef8eaf532871f5e0d3794901fdedde"}
Oct 30 20:44:28.290234 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:28.290142    5958 scope.go:122] "RemoveContainer" containerID="b64d85b9cf28380e9be1c2b2c4e1a0aece7e01b10a0113749246123f4b3e3cdb"
Oct 30 20:44:28.299489 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:28.299443    5958 scope.go:122] "RemoveContainer" containerID="d01a53dd998785d0c615068e1fa68ec12a034e9ad9a06f0e9352e68e65c850a0"
Oct 30 20:44:28.307670 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:28.307648    5958 scope.go:122] "RemoveContainer" containerID="b64d85b9cf28380e9be1c2b2c4e1a0aece7e01b10a0113749246123f4b3e3cdb"
Oct 30 20:44:28.307991 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:44:28.307968    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b64d85b9cf28380e9be1c2b2c4e1a0aece7e01b10a0113749246123f4b3e3cdb\": container with ID starting with b64d85b9cf28380e9be1c2b2c4e1a0aece7e01b10a0113749246123f4b3e3cdb not found: ID does not exist" containerID="b64d85b9cf28380e9be1c2b2c4e1a0aece7e01b10a0113749246123f4b3e3cdb"
Oct 30 20:44:28.308068 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:28.308003    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b64d85b9cf28380e9be1c2b2c4e1a0aece7e01b10a0113749246123f4b3e3cdb"} err="failed to get container status \"b64d85b9cf28380e9be1c2b2c4e1a0aece7e01b10a0113749246123f4b3e3cdb\": rpc error: code = NotFound desc = could not find container \"b64d85b9cf28380e9be1c2b2c4e1a0aece7e01b10a0113749246123f4b3e3cdb\": container with ID starting with b64d85b9cf28380e9be1c2b2c4e1a0aece7e01b10a0113749246123f4b3e3cdb not found: ID does not exist"
Oct 30 20:44:28.308068 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:28.308026    5958 scope.go:122] "RemoveContainer" containerID="d01a53dd998785d0c615068e1fa68ec12a034e9ad9a06f0e9352e68e65c850a0"
Oct 30 20:44:28.308291 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:44:28.308271    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d01a53dd998785d0c615068e1fa68ec12a034e9ad9a06f0e9352e68e65c850a0\": container with ID starting with d01a53dd998785d0c615068e1fa68ec12a034e9ad9a06f0e9352e68e65c850a0 not found: ID does not exist" containerID="d01a53dd998785d0c615068e1fa68ec12a034e9ad9a06f0e9352e68e65c850a0"
Oct 30 20:44:28.308334 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:28.308299    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d01a53dd998785d0c615068e1fa68ec12a034e9ad9a06f0e9352e68e65c850a0"} err="failed to get container status \"d01a53dd998785d0c615068e1fa68ec12a034e9ad9a06f0e9352e68e65c850a0\": rpc error: code = NotFound desc = could not find container \"d01a53dd998785d0c615068e1fa68ec12a034e9ad9a06f0e9352e68e65c850a0\": container with ID starting with d01a53dd998785d0c615068e1fa68ec12a034e9ad9a06f0e9352e68e65c850a0 not found: ID does not exist"
Oct 30 20:44:28.462609 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:28.458220    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"]
Oct 30 20:44:28.505984 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:28.505947    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c530ce-openshift-default-679f95d5c7lqvtq"]
Oct 30 20:44:29.397277 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:29.397242    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7a3317e-2347-410b-a308-496d6ed9b2d7" path="/var/lib/kubelet/pods/c7a3317e-2347-410b-a308-496d6ed9b2d7/volumes"
Oct 30 20:44:30.255711 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.255679    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"]
Oct 30 20:44:30.256138 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.256121    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c7a3317e-2347-410b-a308-496d6ed9b2d7" containerName="istio-proxy"
Oct 30 20:44:30.256194 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.256141    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a3317e-2347-410b-a308-496d6ed9b2d7" containerName="istio-proxy"
Oct 30 20:44:30.256194 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.256161    5958 cpu_manager.go:414] "containerMap: removing container" podUID="327b42ff-e064-4adc-b764-dda3a7afde15" containerName="maas-api"
Oct 30 20:44:30.256194 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.256166    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="327b42ff-e064-4adc-b764-dda3a7afde15" containerName="maas-api"
Oct 30 20:44:30.256194 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.256175    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c7a3317e-2347-410b-a308-496d6ed9b2d7" containerName="istio-proxy"
Oct 30 20:44:30.256194 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.256181    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a3317e-2347-410b-a308-496d6ed9b2d7" containerName="istio-proxy"
Oct 30 20:44:30.256445 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.256262    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="327b42ff-e064-4adc-b764-dda3a7afde15" containerName="maas-api"
Oct 30 20:44:30.256445 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.256272    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a3317e-2347-410b-a308-496d6ed9b2d7" containerName="istio-proxy"
Oct 30 20:44:30.260766 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.260749    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:44:30.292442 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.292405    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-home\") pod \"e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp\" (UID: \"6228f70c-47e8-4e38-bff4-554be632356c\") " pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:44:30.292599 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.292473    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6228f70c-47e8-4e38-bff4-554be632356c-tls-certs\") pod \"e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp\" (UID: \"6228f70c-47e8-4e38-bff4-554be632356c\") " pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:44:30.292646 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.292598    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-model-cache\") pod \"e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp\" (UID: \"6228f70c-47e8-4e38-bff4-554be632356c\") " pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:44:30.292682 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.292646    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhq8m\" (UniqueName: \"kubernetes.io/projected/6228f70c-47e8-4e38-bff4-554be632356c-kube-api-access-zhq8m\") pod \"e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp\" (UID: \"6228f70c-47e8-4e38-bff4-554be632356c\") " pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:44:30.292727 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.292704    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-tmp-dir\") pod \"e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp\" (UID: \"6228f70c-47e8-4e38-bff4-554be632356c\") " pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:44:30.292817 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.292801    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-dshm\") pod \"e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp\" (UID: \"6228f70c-47e8-4e38-bff4-554be632356c\") " pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:44:30.363131 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.363101    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-5291931c-kserve-self-signed-certs\""
Oct 30 20:44:30.364731 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.364703    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"]
Oct 30 20:44:30.394071 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.394047    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-home\") pod \"e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp\" (UID: \"6228f70c-47e8-4e38-bff4-554be632356c\") " pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:44:30.394209 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.394083    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6228f70c-47e8-4e38-bff4-554be632356c-tls-certs\") pod \"e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp\" (UID: \"6228f70c-47e8-4e38-bff4-554be632356c\") " pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:44:30.394209 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.394115    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-model-cache\") pod \"e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp\" (UID: \"6228f70c-47e8-4e38-bff4-554be632356c\") " pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:44:30.394209 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.394154    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhq8m\" (UniqueName: \"kubernetes.io/projected/6228f70c-47e8-4e38-bff4-554be632356c-kube-api-access-zhq8m\") pod \"e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp\" (UID: \"6228f70c-47e8-4e38-bff4-554be632356c\") " pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:44:30.394398 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.394210    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-tmp-dir\") pod \"e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp\" (UID: \"6228f70c-47e8-4e38-bff4-554be632356c\") " pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:44:30.394398 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.394264    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-dshm\") pod \"e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp\" (UID: \"6228f70c-47e8-4e38-bff4-554be632356c\") " pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:44:30.394555 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.394504    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-home\") pod \"e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp\" (UID: \"6228f70c-47e8-4e38-bff4-554be632356c\") " pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:44:30.394830 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.394804    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-tmp-dir\") pod \"e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp\" (UID: \"6228f70c-47e8-4e38-bff4-554be632356c\") " pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:44:30.394830 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.394821    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-model-cache\") pod \"e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp\" (UID: \"6228f70c-47e8-4e38-bff4-554be632356c\") " pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:44:30.396678 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.396650    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-dshm\") pod \"e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp\" (UID: \"6228f70c-47e8-4e38-bff4-554be632356c\") " pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:44:30.397072 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.397026    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6228f70c-47e8-4e38-bff4-554be632356c-tls-certs\") pod \"e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp\" (UID: \"6228f70c-47e8-4e38-bff4-554be632356c\") " pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:44:30.456759 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.456718    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhq8m\" (UniqueName: \"kubernetes.io/projected/6228f70c-47e8-4e38-bff4-554be632356c-kube-api-access-zhq8m\") pod \"e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp\" (UID: \"6228f70c-47e8-4e38-bff4-554be632356c\") " pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:44:30.572308 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.572282    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:44:30.945527 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:30.945500    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"]
Oct 30 20:44:30.947573 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:44:30.947545    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6228f70c_47e8_4e38_bff4_554be632356c.slice/crio-597f8f4106ed2db66f906c3f82c539cf5b142c57d6365e78a48488172f2bc946 WatchSource:0}: Error finding container 597f8f4106ed2db66f906c3f82c539cf5b142c57d6365e78a48488172f2bc946: Status 404 returned error can't find the container with id 597f8f4106ed2db66f906c3f82c539cf5b142c57d6365e78a48488172f2bc946
Oct 30 20:44:31.313196 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:31.313120    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp" event={"ID":"6228f70c-47e8-4e38-bff4-554be632356c","Type":"ContainerStarted","Data":"2358dc4266884f9504e747bf800cbf8a64fc03043074b714043357eddaf611d0"}
Oct 30 20:44:31.313196 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:31.313164    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp" event={"ID":"6228f70c-47e8-4e38-bff4-554be632356c","Type":"ContainerStarted","Data":"597f8f4106ed2db66f906c3f82c539cf5b142c57d6365e78a48488172f2bc946"}
Oct 30 20:44:31.432787 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:31.432729    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp" podStartSLOduration=1.4327125 podStartE2EDuration="1.4327125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:31.43042876 +0000 UTC m=+3710.162644151" watchObservedRunningTime="2026-10-30 20:44:31.4327125 +0000 UTC m=+3710.164927891"
Oct 30 20:44:31.572523 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:31.572493    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:44:33.780338 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:33.780304    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c" podUID="4ebc5902-30bc-4d36-9514-398fa2345eb7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:44:34.585991 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:34.585964    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c"
Oct 30 20:44:34.631235 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:34.631178    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ebc5902-30bc-4d36-9514-398fa2345eb7-kube-api-access-xbgj9\" (UniqueName: \"kubernetes.io/projected/4ebc5902-30bc-4d36-9514-398fa2345eb7-kube-api-access-xbgj9\") pod \"4ebc5902-30bc-4d36-9514-398fa2345eb7\" (UID: \"4ebc5902-30bc-4d36-9514-398fa2345eb7\") "
Oct 30 20:44:34.631235 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:34.631221    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ebc5902-30bc-4d36-9514-398fa2345eb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ebc5902-30bc-4d36-9514-398fa2345eb7-plugins-config-volume\") pod \"4ebc5902-30bc-4d36-9514-398fa2345eb7\" (UID: \"4ebc5902-30bc-4d36-9514-398fa2345eb7\") "
Oct 30 20:44:34.631658 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:34.631629    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ebc5902-30bc-4d36-9514-398fa2345eb7-plugins-config-volume" pod "4ebc5902-30bc-4d36-9514-398fa2345eb7" (UID: "4ebc5902-30bc-4d36-9514-398fa2345eb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:44:34.633346 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:34.633327    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ebc5902-30bc-4d36-9514-398fa2345eb7-kube-api-access-xbgj9" pod "4ebc5902-30bc-4d36-9514-398fa2345eb7" (UID: "4ebc5902-30bc-4d36-9514-398fa2345eb7"). InnerVolumeSpecName "kube-api-access-xbgj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:44:34.733047 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:34.733006    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbgj9\" (UniqueName: \"kubernetes.io/projected/4ebc5902-30bc-4d36-9514-398fa2345eb7-kube-api-access-xbgj9\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:34.733047 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:34.733035    5958 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ebc5902-30bc-4d36-9514-398fa2345eb7-plugins-config-volume\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:44:35.336922 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:35.336879    5958 generic.go:356] "Generic (PLEG): container finished" podID="4ebc5902-30bc-4d36-9514-398fa2345eb7" containerID="c9d32062f7d84a927c585a2323bb354c72cad76b7694641ef9d3fd0708948c61" exitCode=0
Oct 30 20:44:35.337395 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:35.336943    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c"
Oct 30 20:44:35.337395 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:35.336982    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c" event={"ID":"4ebc5902-30bc-4d36-9514-398fa2345eb7","Type":"ContainerDied","Data":"c9d32062f7d84a927c585a2323bb354c72cad76b7694641ef9d3fd0708948c61"}
Oct 30 20:44:35.337395 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:35.337133    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c" event={"ID":"4ebc5902-30bc-4d36-9514-398fa2345eb7","Type":"ContainerDied","Data":"ff2d3b5e46cb0db0d59fc843f4e0be6e62b66a03abb8b1e438e812e77ae5e2b6"}
Oct 30 20:44:35.337395 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:35.337174    5958 scope.go:122] "RemoveContainer" containerID="c9d32062f7d84a927c585a2323bb354c72cad76b7694641ef9d3fd0708948c61"
Oct 30 20:44:35.349518 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:35.349493    5958 scope.go:122] "RemoveContainer" containerID="c9d32062f7d84a927c585a2323bb354c72cad76b7694641ef9d3fd0708948c61"
Oct 30 20:44:35.350014 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:44:35.349809    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d32062f7d84a927c585a2323bb354c72cad76b7694641ef9d3fd0708948c61\": container with ID starting with c9d32062f7d84a927c585a2323bb354c72cad76b7694641ef9d3fd0708948c61 not found: ID does not exist" containerID="c9d32062f7d84a927c585a2323bb354c72cad76b7694641ef9d3fd0708948c61"
Oct 30 20:44:35.350014 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:35.349845    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d32062f7d84a927c585a2323bb354c72cad76b7694641ef9d3fd0708948c61"} err="failed to get container status \"c9d32062f7d84a927c585a2323bb354c72cad76b7694641ef9d3fd0708948c61\": rpc error: code = NotFound desc = could not find container \"c9d32062f7d84a927c585a2323bb354c72cad76b7694641ef9d3fd0708948c61\": container with ID starting with c9d32062f7d84a927c585a2323bb354c72cad76b7694641ef9d3fd0708948c61 not found: ID does not exist"
Oct 30 20:44:35.506444 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:35.506406    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c"]
Oct 30 20:44:35.514828 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:35.514787    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c530ce-778d954dfnpc7c"]
Oct 30 20:44:36.631810 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:44:36.631774    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:44:37.417836 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:37.417797    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ebc5902-30bc-4d36-9514-398fa2345eb7" path="/var/lib/kubelet/pods/4ebc5902-30bc-4d36-9514-398fa2345eb7/volumes"
Oct 30 20:44:40.573168 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:40.573131    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:44:40.585359 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:40.585331    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:44:41.447139 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:41.447108    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:44:43.143175 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:44:43.143146    5958 scope.go:122] "RemoveContainer" containerID="c595c7e95119cac3a240fa2e526a44027ae39546b9d64fda3ea77c6f6612c4f3"
Oct 30 20:45:00.174991 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:00.174950    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq"]
Oct 30 20:45:00.175419 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:00.175368    5958 cpu_manager.go:414] "containerMap: removing container" podUID="4ebc5902-30bc-4d36-9514-398fa2345eb7" containerName="payload-processing"
Oct 30 20:45:00.175419 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:00.175380    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ebc5902-30bc-4d36-9514-398fa2345eb7" containerName="payload-processing"
Oct 30 20:45:00.175556 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:00.175496    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ebc5902-30bc-4d36-9514-398fa2345eb7" containerName="payload-processing"
Oct 30 20:45:00.175556 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:00.175512    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a3317e-2347-410b-a308-496d6ed9b2d7" containerName="istio-proxy"
Oct 30 20:45:00.178716 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:00.178700    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq"
Oct 30 20:45:00.236113 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:00.236084    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq"]
Oct 30 20:45:00.284056 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:00.284031    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:45:00.284056 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:00.284040    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:45:00.284229 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:00.284063    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:45:00.307826 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:00.307797    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3473e77-f70d-4f25-b339-5ae851266dcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889885-4x2cq\" (UID: \"c3473e77-f70d-4f25-b339-5ae851266dcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq"
Oct 30 20:45:00.307960 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:00.307842    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58bnq\" (UniqueName: \"kubernetes.io/projected/c3473e77-f70d-4f25-b339-5ae851266dcb-kube-api-access-58bnq\") pod \"maas-api-key-cleanup-29889885-4x2cq\" (UID: \"c3473e77-f70d-4f25-b339-5ae851266dcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq"
Oct 30 20:45:00.408239 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:00.408214    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3473e77-f70d-4f25-b339-5ae851266dcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889885-4x2cq\" (UID: \"c3473e77-f70d-4f25-b339-5ae851266dcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq"
Oct 30 20:45:00.408361 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:00.408258    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58bnq\" (UniqueName: \"kubernetes.io/projected/c3473e77-f70d-4f25-b339-5ae851266dcb-kube-api-access-58bnq\") pod \"maas-api-key-cleanup-29889885-4x2cq\" (UID: \"c3473e77-f70d-4f25-b339-5ae851266dcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq"
Oct 30 20:45:00.408865 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:00.408847    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3473e77-f70d-4f25-b339-5ae851266dcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889885-4x2cq\" (UID: \"c3473e77-f70d-4f25-b339-5ae851266dcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq"
Oct 30 20:45:00.469730 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:00.469685    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58bnq\" (UniqueName: \"kubernetes.io/projected/c3473e77-f70d-4f25-b339-5ae851266dcb-kube-api-access-58bnq\") pod \"maas-api-key-cleanup-29889885-4x2cq\" (UID: \"c3473e77-f70d-4f25-b339-5ae851266dcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq"
Oct 30 20:45:00.491551 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:00.491532    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq"
Oct 30 20:45:00.864020 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:00.863994    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq"]
Oct 30 20:45:00.866072 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:45:00.866040    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3473e77_f70d_4f25_b339_5ae851266dcb.slice/crio-375d1466ad07a22b4909a5479b99aa7552764e34de2e159e615b6d71e1871062 WatchSource:0}: Error finding container 375d1466ad07a22b4909a5479b99aa7552764e34de2e159e615b6d71e1871062: Status 404 returned error can't find the container with id 375d1466ad07a22b4909a5479b99aa7552764e34de2e159e615b6d71e1871062
Oct 30 20:45:01.534303 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:01.534206    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq" event={"ID":"c3473e77-f70d-4f25-b339-5ae851266dcb","Type":"ContainerStarted","Data":"ad150d0ea4de9b6a7385bb62f8c6d204636242ebaa180377f30169379c705889"}
Oct 30 20:45:01.534303 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:01.534258    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq" event={"ID":"c3473e77-f70d-4f25-b339-5ae851266dcb","Type":"ContainerStarted","Data":"375d1466ad07a22b4909a5479b99aa7552764e34de2e159e615b6d71e1871062"}
Oct 30 20:45:01.651840 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:01.651789    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq" podStartSLOduration=1.65177353 podStartE2EDuration="1.65177353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:01.650402467 +0000 UTC m=+3740.382617856" watchObservedRunningTime="2026-10-30 20:45:01.65177353 +0000 UTC m=+3740.383988920"
Oct 30 20:45:02.540945 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:02.540909    5958 generic.go:356] "Generic (PLEG): container finished" podID="c3473e77-f70d-4f25-b339-5ae851266dcb" containerID="ad150d0ea4de9b6a7385bb62f8c6d204636242ebaa180377f30169379c705889" exitCode=60
Oct 30 20:45:02.541332 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:02.540991    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq" event={"ID":"c3473e77-f70d-4f25-b339-5ae851266dcb","Type":"ContainerDied","Data":"ad150d0ea4de9b6a7385bb62f8c6d204636242ebaa180377f30169379c705889"}
Oct 30 20:45:02.541332 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:02.541155    5958 scope.go:122] "RemoveContainer" containerID="ad150d0ea4de9b6a7385bb62f8c6d204636242ebaa180377f30169379c705889"
Oct 30 20:45:03.547137 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:03.547076    5958 generic.go:356] "Generic (PLEG): container finished" podID="c3473e77-f70d-4f25-b339-5ae851266dcb" containerID="5063433bde62ab892d8269a6ea594a722a63c12b038488750eb6ab85dd46ecc1" exitCode=60
Oct 30 20:45:03.547546 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:03.547141    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq" event={"ID":"c3473e77-f70d-4f25-b339-5ae851266dcb","Type":"ContainerDied","Data":"5063433bde62ab892d8269a6ea594a722a63c12b038488750eb6ab85dd46ecc1"}
Oct 30 20:45:03.547546 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:03.547189    5958 scope.go:122] "RemoveContainer" containerID="ad150d0ea4de9b6a7385bb62f8c6d204636242ebaa180377f30169379c705889"
Oct 30 20:45:03.547546 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:03.547327    5958 scope.go:122] "RemoveContainer" containerID="5063433bde62ab892d8269a6ea594a722a63c12b038488750eb6ab85dd46ecc1"
Oct 30 20:45:03.547721 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:45:03.547577    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889885-4x2cq_odh-ai-gateway-infra(c3473e77-f70d-4f25-b339-5ae851266dcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq" podUID="c3473e77-f70d-4f25-b339-5ae851266dcb"
Oct 30 20:45:04.553601 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:04.553572    5958 scope.go:122] "RemoveContainer" containerID="5063433bde62ab892d8269a6ea594a722a63c12b038488750eb6ab85dd46ecc1"
Oct 30 20:45:04.553974 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:45:04.553769    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889885-4x2cq_odh-ai-gateway-infra(c3473e77-f70d-4f25-b339-5ae851266dcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq" podUID="c3473e77-f70d-4f25-b339-5ae851266dcb"
Oct 30 20:45:06.604796 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:45:06.604758    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:45:10.318407 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:10.318370    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"]
Oct 30 20:45:10.323357 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:10.323328    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:45:10.398683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:10.398650    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-tls-certs\") pod \"e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4\" (UID: \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\") " pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:45:10.398683 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:10.398692    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-dshm\") pod \"e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4\" (UID: \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\") " pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:45:10.398898 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:10.398727    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-tmp-dir\") pod \"e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4\" (UID: \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\") " pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:45:10.398898 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:10.398851    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfcml\" (UniqueName: \"kubernetes.io/projected/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-kube-api-access-bfcml\") pod \"e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4\" (UID: \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\") " pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:45:10.398997 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:10.398899    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-model-cache\") pod \"e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4\" (UID: \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\") " pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:45:10.398997 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:10.398981    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-home\") pod \"e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4\" (UID: \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\") " pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:45:10.425784 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:10.425756    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-5291931c-kserve-self-signed-certs\""
Oct 30 20:45:10.427471 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:10.427438    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"]
Oct 30 20:45:10.500389 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:10.500353    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-home\") pod \"e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4\" (UID: \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\") " pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:45:10.500619 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:10.500434    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-tls-certs\") pod \"e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4\" (UID: \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\") " pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:45:10.500619 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:10.500478    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-dshm\") pod \"e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4\" (UID: \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\") " pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:45:10.500619 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:10.500499    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-tmp-dir\") pod \"e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4\" (UID: \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\") " pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:45:10.500619 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:10.500533    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfcml\" (UniqueName: \"kubernetes.io/projected/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-kube-api-access-bfcml\") pod \"e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4\" (UID: \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\") " pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:45:10.500619 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:10.500571    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-model-cache\") pod \"e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4\" (UID: \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\") " pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:45:10.500909 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:10.500834    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-home\") pod \"e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4\" (UID: \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\") " pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:45:10.500971 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:10.500911    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-model-cache\") pod \"e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4\" (UID: \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\") " pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:45:10.500971 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:10.500949    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-tmp-dir\") pod \"e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4\" (UID: \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\") " pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:45:10.502795 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:10.502775    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-dshm\") pod \"e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4\" (UID: \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\") " pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:45:10.503063 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:10.503045    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-tls-certs\") pod \"e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4\" (UID: \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\") " pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:45:10.561842 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:10.561812    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfcml\" (UniqueName: \"kubernetes.io/projected/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-kube-api-access-bfcml\") pod \"e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4\" (UID: \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\") " pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:45:10.634463 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:10.634385    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:45:11.015540 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:11.015515    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"]
Oct 30 20:45:11.017328 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:45:11.017303    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1dc7f07_65ea_459e_b5ce_02fe26c9d193.slice/crio-2272f5db01c18b13d09e249fdbfbf7230a8f455d3235149eb127ab454fb522ea WatchSource:0}: Error finding container 2272f5db01c18b13d09e249fdbfbf7230a8f455d3235149eb127ab454fb522ea: Status 404 returned error can't find the container with id 2272f5db01c18b13d09e249fdbfbf7230a8f455d3235149eb127ab454fb522ea
Oct 30 20:45:11.587494 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:11.587445    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4" event={"ID":"f1dc7f07-65ea-459e-b5ce-02fe26c9d193","Type":"ContainerStarted","Data":"f3d3f1c5cf54831dcc638c7f86bc2bed4709b7437850b9d8b7d66ead428bf546"}
Oct 30 20:45:11.587494 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:11.587497    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4" event={"ID":"f1dc7f07-65ea-459e-b5ce-02fe26c9d193","Type":"ContainerStarted","Data":"2272f5db01c18b13d09e249fdbfbf7230a8f455d3235149eb127ab454fb522ea"}
Oct 30 20:45:11.634800 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:11.634776    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:45:11.706817 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:11.706776    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4" podStartSLOduration=1.7067643860000001 podStartE2EDuration="1.706764386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:11.705511604 +0000 UTC m=+3750.437726994" watchObservedRunningTime="2026-10-30 20:45:11.706764386 +0000 UTC m=+3750.438979776"
Oct 30 20:45:14.391831 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:14.391799    5958 scope.go:122] "RemoveContainer" containerID="5063433bde62ab892d8269a6ea594a722a63c12b038488750eb6ab85dd46ecc1"
Oct 30 20:45:15.612382 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:15.612342    5958 generic.go:356] "Generic (PLEG): container finished" podID="c3473e77-f70d-4f25-b339-5ae851266dcb" containerID="e29ad3e86f44fc7bc79487b080abd5d83fde6cd916a7d66d8fc6bba44e2ff623" exitCode=60
Oct 30 20:45:15.612931 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:15.612430    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq" event={"ID":"c3473e77-f70d-4f25-b339-5ae851266dcb","Type":"ContainerDied","Data":"e29ad3e86f44fc7bc79487b080abd5d83fde6cd916a7d66d8fc6bba44e2ff623"}
Oct 30 20:45:15.612931 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:15.612532    5958 scope.go:122] "RemoveContainer" containerID="5063433bde62ab892d8269a6ea594a722a63c12b038488750eb6ab85dd46ecc1"
Oct 30 20:45:15.612931 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:15.612919    5958 scope.go:122] "RemoveContainer" containerID="e29ad3e86f44fc7bc79487b080abd5d83fde6cd916a7d66d8fc6bba44e2ff623"
Oct 30 20:45:15.613252 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:45:15.613226    5958 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889885-4x2cq_odh-ai-gateway-infra(c3473e77-f70d-4f25-b339-5ae851266dcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq" podUID="c3473e77-f70d-4f25-b339-5ae851266dcb"
Oct 30 20:45:16.745607 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:16.745566    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq"]
Oct 30 20:45:17.126605 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:17.126263    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq"
Oct 30 20:45:17.323732 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:17.309768    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3473e77-f70d-4f25-b339-5ae851266dcb-kube-api-access-58bnq\" (UniqueName: \"kubernetes.io/projected/c3473e77-f70d-4f25-b339-5ae851266dcb-kube-api-access-58bnq\") pod \"c3473e77-f70d-4f25-b339-5ae851266dcb\" (UID: \"c3473e77-f70d-4f25-b339-5ae851266dcb\") "
Oct 30 20:45:17.323732 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:17.309866    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3473e77-f70d-4f25-b339-5ae851266dcb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3473e77-f70d-4f25-b339-5ae851266dcb-maas-api-service-ca\") pod \"c3473e77-f70d-4f25-b339-5ae851266dcb\" (UID: \"c3473e77-f70d-4f25-b339-5ae851266dcb\") "
Oct 30 20:45:17.323732 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:17.323635    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3473e77-f70d-4f25-b339-5ae851266dcb-maas-api-service-ca" pod "c3473e77-f70d-4f25-b339-5ae851266dcb" (UID: "c3473e77-f70d-4f25-b339-5ae851266dcb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:45:17.324047 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:17.323972    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3473e77-f70d-4f25-b339-5ae851266dcb-kube-api-access-58bnq" pod "c3473e77-f70d-4f25-b339-5ae851266dcb" (UID: "c3473e77-f70d-4f25-b339-5ae851266dcb"). InnerVolumeSpecName "kube-api-access-58bnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:45:17.414479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:17.411086    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58bnq\" (UniqueName: \"kubernetes.io/projected/c3473e77-f70d-4f25-b339-5ae851266dcb-kube-api-access-58bnq\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:17.414479 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:17.411123    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3473e77-f70d-4f25-b339-5ae851266dcb-maas-api-service-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:17.526537 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:45:17.526275    5958 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3473e77_f70d_4f25_b339_5ae851266dcb.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 20:45:17.630748 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:17.630668    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq"
Oct 30 20:45:17.630748 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:17.630697    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq" event={"ID":"c3473e77-f70d-4f25-b339-5ae851266dcb","Type":"ContainerDied","Data":"375d1466ad07a22b4909a5479b99aa7552764e34de2e159e615b6d71e1871062"}
Oct 30 20:45:17.630954 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:17.630774    5958 scope.go:122] "RemoveContainer" containerID="e29ad3e86f44fc7bc79487b080abd5d83fde6cd916a7d66d8fc6bba44e2ff623"
Oct 30 20:45:17.807474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:17.806623    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq"]
Oct 30 20:45:17.857476 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:17.857387    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-4x2cq"]
Oct 30 20:45:19.397733 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:19.397701    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3473e77-f70d-4f25-b339-5ae851266dcb" path="/var/lib/kubelet/pods/c3473e77-f70d-4f25-b339-5ae851266dcb/volumes"
Oct 30 20:45:20.634971 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:20.634938    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:45:20.648200 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:20.648167    5958 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:45:20.659238 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:20.659214    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:45:24.059467 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:24.059407    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-chfp4"]
Oct 30 20:45:24.109581 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:24.109552    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-chfp4"]
Oct 30 20:45:25.397596 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:25.397497    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88a0acb6-3ae8-44bf-bbf0-cf559290357e" path="/var/lib/kubelet/pods/88a0acb6-3ae8-44bf-bbf0-cf559290357e/volumes"
Oct 30 20:45:36.604616 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:45:36.604579    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:45:41.671619 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:41.671571    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bgls7"]
Oct 30 20:45:41.672042 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:41.672028    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c3473e77-f70d-4f25-b339-5ae851266dcb" containerName="cleanup"
Oct 30 20:45:41.672095 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:41.672044    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3473e77-f70d-4f25-b339-5ae851266dcb" containerName="cleanup"
Oct 30 20:45:41.672095 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:41.672084    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c3473e77-f70d-4f25-b339-5ae851266dcb" containerName="cleanup"
Oct 30 20:45:41.672095 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:41.672090    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3473e77-f70d-4f25-b339-5ae851266dcb" containerName="cleanup"
Oct 30 20:45:41.672194 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:41.672170    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3473e77-f70d-4f25-b339-5ae851266dcb" containerName="cleanup"
Oct 30 20:45:41.672194 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:41.672184    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3473e77-f70d-4f25-b339-5ae851266dcb" containerName="cleanup"
Oct 30 20:45:41.676563 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:41.676538    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bgls7"
Oct 30 20:45:41.759187 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:41.759152    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48bfa585-2d57-469c-ba55-8ac07b7100c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-bgls7\" (UID: \"48bfa585-2d57-469c-ba55-8ac07b7100c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bgls7"
Oct 30 20:45:41.759324 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:41.759215    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5gbq\" (UniqueName: \"kubernetes.io/projected/48bfa585-2d57-469c-ba55-8ac07b7100c5-kube-api-access-d5gbq\") pod \"maas-api-revoke-sub-452b6d7b-bgls7\" (UID: \"48bfa585-2d57-469c-ba55-8ac07b7100c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bgls7"
Oct 30 20:45:41.759438 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:41.759396    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48bfa585-2d57-469c-ba55-8ac07b7100c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-bgls7\" (UID: \"48bfa585-2d57-469c-ba55-8ac07b7100c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bgls7"
Oct 30 20:45:41.780994 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:41.780970    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:45:41.781133 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:41.781002    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:45:41.781133 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:41.781042    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:45:41.781133 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:41.781066    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:45:41.782637 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:41.782618    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bgls7"]
Oct 30 20:45:41.860229 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:41.860206    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48bfa585-2d57-469c-ba55-8ac07b7100c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-bgls7\" (UID: \"48bfa585-2d57-469c-ba55-8ac07b7100c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bgls7"
Oct 30 20:45:41.860355 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:41.860240    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5gbq\" (UniqueName: \"kubernetes.io/projected/48bfa585-2d57-469c-ba55-8ac07b7100c5-kube-api-access-d5gbq\") pod \"maas-api-revoke-sub-452b6d7b-bgls7\" (UID: \"48bfa585-2d57-469c-ba55-8ac07b7100c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bgls7"
Oct 30 20:45:41.860355 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:41.860291    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48bfa585-2d57-469c-ba55-8ac07b7100c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-bgls7\" (UID: \"48bfa585-2d57-469c-ba55-8ac07b7100c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bgls7"
Oct 30 20:45:41.860897 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:41.860879    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48bfa585-2d57-469c-ba55-8ac07b7100c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-bgls7\" (UID: \"48bfa585-2d57-469c-ba55-8ac07b7100c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bgls7"
Oct 30 20:45:41.862717 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:41.862697    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48bfa585-2d57-469c-ba55-8ac07b7100c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-bgls7\" (UID: \"48bfa585-2d57-469c-ba55-8ac07b7100c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bgls7"
Oct 30 20:45:41.921793 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:41.921736    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5gbq\" (UniqueName: \"kubernetes.io/projected/48bfa585-2d57-469c-ba55-8ac07b7100c5-kube-api-access-d5gbq\") pod \"maas-api-revoke-sub-452b6d7b-bgls7\" (UID: \"48bfa585-2d57-469c-ba55-8ac07b7100c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bgls7"
Oct 30 20:45:41.990055 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:41.990037    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bgls7"
Oct 30 20:45:42.360963 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:42.360938    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bgls7"]
Oct 30 20:45:42.363498 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:45:42.363476    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48bfa585_2d57_469c_ba55_8ac07b7100c5.slice/crio-5cbf6a55844e005b7b6f41ab8a2ed460de88c7e28986433b6dc053261a45e56f WatchSource:0}: Error finding container 5cbf6a55844e005b7b6f41ab8a2ed460de88c7e28986433b6dc053261a45e56f: Status 404 returned error can't find the container with id 5cbf6a55844e005b7b6f41ab8a2ed460de88c7e28986433b6dc053261a45e56f
Oct 30 20:45:42.763859 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:42.763831    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bgls7" event={"ID":"48bfa585-2d57-469c-ba55-8ac07b7100c5","Type":"ContainerStarted","Data":"8000716522f08d81bc73965969e4281f9ef51d55541440f9f42bcf9e429a74e1"}
Oct 30 20:45:42.764341 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:42.763892    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bgls7" event={"ID":"48bfa585-2d57-469c-ba55-8ac07b7100c5","Type":"ContainerStarted","Data":"5cbf6a55844e005b7b6f41ab8a2ed460de88c7e28986433b6dc053261a45e56f"}
Oct 30 20:45:42.881995 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:42.881946    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bgls7" podStartSLOduration=1.88193338 podStartE2EDuration="1.88193338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:42.880402513 +0000 UTC m=+3781.612617903" watchObservedRunningTime="2026-10-30 20:45:42.88193338 +0000 UTC m=+3781.614148769"
Oct 30 20:45:43.772872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:43.772826    5958 generic.go:356] "Generic (PLEG): container finished" podID="48bfa585-2d57-469c-ba55-8ac07b7100c5" containerID="8000716522f08d81bc73965969e4281f9ef51d55541440f9f42bcf9e429a74e1" exitCode=0
Oct 30 20:45:43.772872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:43.772875    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bgls7" event={"ID":"48bfa585-2d57-469c-ba55-8ac07b7100c5","Type":"ContainerDied","Data":"8000716522f08d81bc73965969e4281f9ef51d55541440f9f42bcf9e429a74e1"}
Oct 30 20:45:45.182953 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:45.182617    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bgls7"
Oct 30 20:45:45.221499 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:45.221469    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48bfa585-2d57-469c-ba55-8ac07b7100c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48bfa585-2d57-469c-ba55-8ac07b7100c5-maas-api-xks-ca\") pod \"48bfa585-2d57-469c-ba55-8ac07b7100c5\" (UID: \"48bfa585-2d57-469c-ba55-8ac07b7100c5\") "
Oct 30 20:45:45.221635 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:45.221578    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48bfa585-2d57-469c-ba55-8ac07b7100c5-kube-api-access-d5gbq\" (UniqueName: \"kubernetes.io/projected/48bfa585-2d57-469c-ba55-8ac07b7100c5-kube-api-access-d5gbq\") pod \"48bfa585-2d57-469c-ba55-8ac07b7100c5\" (UID: \"48bfa585-2d57-469c-ba55-8ac07b7100c5\") "
Oct 30 20:45:45.221708 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:45.221653    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48bfa585-2d57-469c-ba55-8ac07b7100c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48bfa585-2d57-469c-ba55-8ac07b7100c5-maas-api-openshift-ca\") pod \"48bfa585-2d57-469c-ba55-8ac07b7100c5\" (UID: \"48bfa585-2d57-469c-ba55-8ac07b7100c5\") "
Oct 30 20:45:45.222046 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:45.222023    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48bfa585-2d57-469c-ba55-8ac07b7100c5-maas-api-openshift-ca" pod "48bfa585-2d57-469c-ba55-8ac07b7100c5" (UID: "48bfa585-2d57-469c-ba55-8ac07b7100c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:45:45.223836 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:45.223812    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48bfa585-2d57-469c-ba55-8ac07b7100c5-maas-api-xks-ca" pod "48bfa585-2d57-469c-ba55-8ac07b7100c5" (UID: "48bfa585-2d57-469c-ba55-8ac07b7100c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:45:45.223953 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:45.223938    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48bfa585-2d57-469c-ba55-8ac07b7100c5-kube-api-access-d5gbq" pod "48bfa585-2d57-469c-ba55-8ac07b7100c5" (UID: "48bfa585-2d57-469c-ba55-8ac07b7100c5"). InnerVolumeSpecName "kube-api-access-d5gbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:45:45.322271 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:45.322238    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48bfa585-2d57-469c-ba55-8ac07b7100c5-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:45.322271 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:45.322268    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48bfa585-2d57-469c-ba55-8ac07b7100c5-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:45.322271 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:45.322279    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5gbq\" (UniqueName: \"kubernetes.io/projected/48bfa585-2d57-469c-ba55-8ac07b7100c5-kube-api-access-d5gbq\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:45:45.804705 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:45.804620    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bgls7" event={"ID":"48bfa585-2d57-469c-ba55-8ac07b7100c5","Type":"ContainerDied","Data":"5cbf6a55844e005b7b6f41ab8a2ed460de88c7e28986433b6dc053261a45e56f"}
Oct 30 20:45:45.808557 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:45.808531    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cbf6a55844e005b7b6f41ab8a2ed460de88c7e28986433b6dc053261a45e56f"
Oct 30 20:45:45.808732 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:45.807588    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5295k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:45:45.808887 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:45.808864    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k" podUID="cc591cde-0da2-429d-8b99-a98f35abadd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:45:45.809258 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:45.806574    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bgls7"
Oct 30 20:45:54.695245 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:45:54.695164    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"]
Oct 30 20:46:03.870717 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:03.870683    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"]
Oct 30 20:46:06.604654 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:46:06.604604    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:46:09.715174 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:09.715129    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp" podUID="6228f70c-47e8-4e38-bff4-554be632356c" containerName="main" containerID="cri-o://2358dc4266884f9504e747bf800cbf8a64fc03043074b714043357eddaf611d0" gracePeriod=45
Oct 30 20:46:09.900510 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:09.900474    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6qmsf"]
Oct 30 20:46:09.901190 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:09.901163    5958 cpu_manager.go:414] "containerMap: removing container" podUID="48bfa585-2d57-469c-ba55-8ac07b7100c5" containerName="revoke-keys"
Oct 30 20:46:09.901328 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:09.901193    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48bfa585-2d57-469c-ba55-8ac07b7100c5" containerName="revoke-keys"
Oct 30 20:46:09.901328 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:09.901243    5958 cpu_manager.go:414] "containerMap: removing container" podUID="c3473e77-f70d-4f25-b339-5ae851266dcb" containerName="cleanup"
Oct 30 20:46:09.901328 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:09.901251    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3473e77-f70d-4f25-b339-5ae851266dcb" containerName="cleanup"
Oct 30 20:46:09.901504 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:09.901377    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="48bfa585-2d57-469c-ba55-8ac07b7100c5" containerName="revoke-keys"
Oct 30 20:46:09.901504 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:09.901403    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3473e77-f70d-4f25-b339-5ae851266dcb" containerName="cleanup"
Oct 30 20:46:09.905144 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:09.905120    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6qmsf"
Oct 30 20:46:09.950061 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:09.950028    5958 generic.go:356] "Generic (PLEG): container finished" podID="6228f70c-47e8-4e38-bff4-554be632356c" containerID="2358dc4266884f9504e747bf800cbf8a64fc03043074b714043357eddaf611d0" exitCode=0
Oct 30 20:46:09.950165 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:09.950097    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp" event={"ID":"6228f70c-47e8-4e38-bff4-554be632356c","Type":"ContainerDied","Data":"2358dc4266884f9504e747bf800cbf8a64fc03043074b714043357eddaf611d0"}
Oct 30 20:46:09.950165 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:09.950145    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp" event={"ID":"6228f70c-47e8-4e38-bff4-554be632356c","Type":"ContainerDied","Data":"597f8f4106ed2db66f906c3f82c539cf5b142c57d6365e78a48488172f2bc946"}
Oct 30 20:46:09.950258 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:09.950178    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="597f8f4106ed2db66f906c3f82c539cf5b142c57d6365e78a48488172f2bc946"
Oct 30 20:46:09.958069 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:09.958049    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:46:09.958225 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:09.958132    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:46:09.958225 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:09.958161    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:46:09.964826 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:09.964803    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:46:10.009939 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.009878    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6qmsf"]
Oct 30 20:46:10.017471 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.017432    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4zdv\" (UniqueName: \"kubernetes.io/projected/ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b-kube-api-access-l4zdv\") pod \"maas-api-revoke-sub-3e31a506-6qmsf\" (UID: \"ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6qmsf"
Oct 30 20:46:10.017598 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.017577    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-6qmsf\" (UID: \"ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6qmsf"
Oct 30 20:46:10.017726 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.017714    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-6qmsf\" (UID: \"ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6qmsf"
Oct 30 20:46:10.058635 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.058616    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:46:10.118694 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.118672    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6228f70c-47e8-4e38-bff4-554be632356c-tls-certs\" (UniqueName: \"kubernetes.io/secret/6228f70c-47e8-4e38-bff4-554be632356c-tls-certs\") pod \"6228f70c-47e8-4e38-bff4-554be632356c\" (UID: \"6228f70c-47e8-4e38-bff4-554be632356c\") "
Oct 30 20:46:10.118784 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.118708    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6228f70c-47e8-4e38-bff4-554be632356c-kube-api-access-zhq8m\" (UniqueName: \"kubernetes.io/projected/6228f70c-47e8-4e38-bff4-554be632356c-kube-api-access-zhq8m\") pod \"6228f70c-47e8-4e38-bff4-554be632356c\" (UID: \"6228f70c-47e8-4e38-bff4-554be632356c\") "
Oct 30 20:46:10.118784 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.118737    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-home\" (UniqueName: \"kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-home\") pod \"6228f70c-47e8-4e38-bff4-554be632356c\" (UID: \"6228f70c-47e8-4e38-bff4-554be632356c\") "
Oct 30 20:46:10.118784 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.118777    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-model-cache\") pod \"6228f70c-47e8-4e38-bff4-554be632356c\" (UID: \"6228f70c-47e8-4e38-bff4-554be632356c\") "
Oct 30 20:46:10.118925 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.118808    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-tmp-dir\") pod \"6228f70c-47e8-4e38-bff4-554be632356c\" (UID: \"6228f70c-47e8-4e38-bff4-554be632356c\") "
Oct 30 20:46:10.118925 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.118832    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-dshm\") pod \"6228f70c-47e8-4e38-bff4-554be632356c\" (UID: \"6228f70c-47e8-4e38-bff4-554be632356c\") "
Oct 30 20:46:10.119032 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.118982    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-6qmsf\" (UID: \"ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6qmsf"
Oct 30 20:46:10.119032 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.119024    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4zdv\" (UniqueName: \"kubernetes.io/projected/ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b-kube-api-access-l4zdv\") pod \"maas-api-revoke-sub-3e31a506-6qmsf\" (UID: \"ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6qmsf"
Oct 30 20:46:10.119143 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.119072    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-model-cache" pod "6228f70c-47e8-4e38-bff4-554be632356c" (UID: "6228f70c-47e8-4e38-bff4-554be632356c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:46:10.119143 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.119087    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-home" pod "6228f70c-47e8-4e38-bff4-554be632356c" (UID: "6228f70c-47e8-4e38-bff4-554be632356c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:46:10.119143 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.119109    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-tmp-dir" pod "6228f70c-47e8-4e38-bff4-554be632356c" (UID: "6228f70c-47e8-4e38-bff4-554be632356c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:46:10.119284 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.119162    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-6qmsf\" (UID: \"ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6qmsf"
Oct 30 20:46:10.119284 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.119265    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-home\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:10.119414 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.119281    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-model-cache\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:10.119564 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.119324    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-tmp-dir\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:10.120094 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.120068    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-6qmsf\" (UID: \"ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6qmsf"
Oct 30 20:46:10.121175 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.121149    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6228f70c-47e8-4e38-bff4-554be632356c-tls-certs" pod "6228f70c-47e8-4e38-bff4-554be632356c" (UID: "6228f70c-47e8-4e38-bff4-554be632356c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:46:10.121271 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.121220    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6228f70c-47e8-4e38-bff4-554be632356c-kube-api-access-zhq8m" pod "6228f70c-47e8-4e38-bff4-554be632356c" (UID: "6228f70c-47e8-4e38-bff4-554be632356c"). InnerVolumeSpecName "kube-api-access-zhq8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:46:10.121318 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.121302    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-dshm" pod "6228f70c-47e8-4e38-bff4-554be632356c" (UID: "6228f70c-47e8-4e38-bff4-554be632356c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:46:10.121462 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.121430    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-6qmsf\" (UID: \"ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6qmsf"
Oct 30 20:46:10.180217 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.180197    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4zdv\" (UniqueName: \"kubernetes.io/projected/ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b-kube-api-access-l4zdv\") pod \"maas-api-revoke-sub-3e31a506-6qmsf\" (UID: \"ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6qmsf"
Oct 30 20:46:10.219543 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.219522    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6qmsf"
Oct 30 20:46:10.220902 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.220882    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6228f70c-47e8-4e38-bff4-554be632356c-tls-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:10.220979 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.220907    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhq8m\" (UniqueName: \"kubernetes.io/projected/6228f70c-47e8-4e38-bff4-554be632356c-kube-api-access-zhq8m\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:10.220979 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.220917    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6228f70c-47e8-4e38-bff4-554be632356c-dshm\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:10.589637 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.589611    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6qmsf"]
Oct 30 20:46:10.591601 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:46:10.591576    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa653c7_81f6_49f8_9f30_d2ffc90a2f2b.slice/crio-f8db9c984f98e283371c6f1dac9a51f3a1a8c5de3fd22b9db64fb2e72d8ed893 WatchSource:0}: Error finding container f8db9c984f98e283371c6f1dac9a51f3a1a8c5de3fd22b9db64fb2e72d8ed893: Status 404 returned error can't find the container with id f8db9c984f98e283371c6f1dac9a51f3a1a8c5de3fd22b9db64fb2e72d8ed893
Oct 30 20:46:10.956822 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.956790    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"
Oct 30 20:46:10.957227 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.956821    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6qmsf" event={"ID":"ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b","Type":"ContainerStarted","Data":"eb6d3f057ade5e9026fea513c4dc1a90c3771beee4d62cc763d53495e1cada7d"}
Oct 30 20:46:10.957227 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:10.956872    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6qmsf" event={"ID":"ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b","Type":"ContainerStarted","Data":"f8db9c984f98e283371c6f1dac9a51f3a1a8c5de3fd22b9db64fb2e72d8ed893"}
Oct 30 20:46:11.074124 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:11.074072    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6qmsf" podStartSLOduration=2.07405679 podStartE2EDuration="2.07405679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:11.073524045 +0000 UTC m=+3809.805739436" watchObservedRunningTime="2026-10-30 20:46:11.07405679 +0000 UTC m=+3809.806272171"
Oct 30 20:46:11.238073 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:11.238003    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"]
Oct 30 20:46:11.241872 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:11.241849    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-5291931c-kserve-75f497449c-kq9rp"]
Oct 30 20:46:11.397203 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:11.397175    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6228f70c-47e8-4e38-bff4-554be632356c" path="/var/lib/kubelet/pods/6228f70c-47e8-4e38-bff4-554be632356c/volumes"
Oct 30 20:46:11.963725 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:11.963695    5958 generic.go:356] "Generic (PLEG): container finished" podID="ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b" containerID="eb6d3f057ade5e9026fea513c4dc1a90c3771beee4d62cc763d53495e1cada7d" exitCode=0
Oct 30 20:46:11.964116 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:11.963780    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6qmsf" event={"ID":"ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b","Type":"ContainerDied","Data":"eb6d3f057ade5e9026fea513c4dc1a90c3771beee4d62cc763d53495e1cada7d"}
Oct 30 20:46:13.100986 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:13.100963    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6qmsf"
Oct 30 20:46:13.249388 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:13.249325    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b-kube-api-access-l4zdv\" (UniqueName: \"kubernetes.io/projected/ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b-kube-api-access-l4zdv\") pod \"ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b\" (UID: \"ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b\") "
Oct 30 20:46:13.249549 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:13.249426    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b-maas-api-openshift-ca\") pod \"ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b\" (UID: \"ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b\") "
Oct 30 20:46:13.249549 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:13.249536    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b-maas-api-xks-ca\") pod \"ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b\" (UID: \"ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b\") "
Oct 30 20:46:13.249824 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:13.249785    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b-maas-api-openshift-ca" pod "ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b" (UID: "ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:46:13.249964 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:13.249947    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:13.251523 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:13.251497    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b-kube-api-access-l4zdv" pod "ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b" (UID: "ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b"). InnerVolumeSpecName "kube-api-access-l4zdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:46:13.251668 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:13.251649    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b-maas-api-xks-ca" pod "ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b" (UID: "ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:46:13.351112 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:13.351089    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4zdv\" (UniqueName: \"kubernetes.io/projected/ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b-kube-api-access-l4zdv\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:13.351112 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:13.351112    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:13.978223 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:13.978194    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6qmsf"
Oct 30 20:46:13.978746 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:13.978232    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6qmsf" event={"ID":"ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b","Type":"ContainerDied","Data":"f8db9c984f98e283371c6f1dac9a51f3a1a8c5de3fd22b9db64fb2e72d8ed893"}
Oct 30 20:46:13.978887 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:13.978773    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8db9c984f98e283371c6f1dac9a51f3a1a8c5de3fd22b9db64fb2e72d8ed893"
Oct 30 20:46:15.822621 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:15.822587    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5295k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:46:15.822996 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:15.822640    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k" podUID="cc591cde-0da2-429d-8b99-a98f35abadd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:46:18.890382 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:18.890323    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4" podUID="f1dc7f07-65ea-459e-b5ce-02fe26c9d193" containerName="main" containerID="cri-o://f3d3f1c5cf54831dcc638c7f86bc2bed4709b7437850b9d8b7d66ead428bf546" gracePeriod=45
Oct 30 20:46:19.016849 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:19.016821    5958 generic.go:356] "Generic (PLEG): container finished" podID="f1dc7f07-65ea-459e-b5ce-02fe26c9d193" containerID="f3d3f1c5cf54831dcc638c7f86bc2bed4709b7437850b9d8b7d66ead428bf546" exitCode=0
Oct 30 20:46:19.016961 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:19.016879    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4" event={"ID":"f1dc7f07-65ea-459e-b5ce-02fe26c9d193","Type":"ContainerDied","Data":"f3d3f1c5cf54831dcc638c7f86bc2bed4709b7437850b9d8b7d66ead428bf546"}
Oct 30 20:46:19.136242 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:19.136220    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:46:19.246070 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:19.246010    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-kube-api-access-bfcml\" (UniqueName: \"kubernetes.io/projected/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-kube-api-access-bfcml\") pod \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\" (UID: \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\") "
Oct 30 20:46:19.246070 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:19.246045    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-tmp-dir\") pod \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\" (UID: \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\") "
Oct 30 20:46:19.246259 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:19.246100    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-tls-certs\" (UniqueName: \"kubernetes.io/secret/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-tls-certs\") pod \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\" (UID: \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\") "
Oct 30 20:46:19.246259 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:19.246119    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-model-cache\") pod \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\" (UID: \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\") "
Oct 30 20:46:19.246259 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:19.246168    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-dshm\") pod \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\" (UID: \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\") "
Oct 30 20:46:19.246259 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:19.246187    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-home\" (UniqueName: \"kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-home\") pod \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\" (UID: \"f1dc7f07-65ea-459e-b5ce-02fe26c9d193\") "
Oct 30 20:46:19.246508 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:19.246364    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-tmp-dir" pod "f1dc7f07-65ea-459e-b5ce-02fe26c9d193" (UID: "f1dc7f07-65ea-459e-b5ce-02fe26c9d193"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:46:19.246508 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:19.246404    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-model-cache" pod "f1dc7f07-65ea-459e-b5ce-02fe26c9d193" (UID: "f1dc7f07-65ea-459e-b5ce-02fe26c9d193"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:46:19.246617 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:19.246545    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-home" pod "f1dc7f07-65ea-459e-b5ce-02fe26c9d193" (UID: "f1dc7f07-65ea-459e-b5ce-02fe26c9d193"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:46:19.246658 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:19.246633    5958 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-model-cache\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:19.246837 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:19.246745    5958 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-tmp-dir\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:19.248180 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:19.248159    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-kube-api-access-bfcml" pod "f1dc7f07-65ea-459e-b5ce-02fe26c9d193" (UID: "f1dc7f07-65ea-459e-b5ce-02fe26c9d193"). InnerVolumeSpecName "kube-api-access-bfcml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:46:19.248738 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:19.248718    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-dshm" pod "f1dc7f07-65ea-459e-b5ce-02fe26c9d193" (UID: "f1dc7f07-65ea-459e-b5ce-02fe26c9d193"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:46:19.248836 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:19.248761    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-tls-certs" pod "f1dc7f07-65ea-459e-b5ce-02fe26c9d193" (UID: "f1dc7f07-65ea-459e-b5ce-02fe26c9d193"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:46:19.347373 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:19.347345    5958 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-tls-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:19.347373 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:19.347372    5958 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-dshm\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:19.347560 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:19.347381    5958 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-home\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:19.347560 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:19.347390    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfcml\" (UniqueName: \"kubernetes.io/projected/f1dc7f07-65ea-459e-b5ce-02fe26c9d193-kube-api-access-bfcml\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:20.023798 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:20.023766    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"
Oct 30 20:46:20.024295 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:20.023799    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4" event={"ID":"f1dc7f07-65ea-459e-b5ce-02fe26c9d193","Type":"ContainerDied","Data":"2272f5db01c18b13d09e249fdbfbf7230a8f455d3235149eb127ab454fb522ea"}
Oct 30 20:46:20.024295 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:20.023844    5958 scope.go:122] "RemoveContainer" containerID="f3d3f1c5cf54831dcc638c7f86bc2bed4709b7437850b9d8b7d66ead428bf546"
Oct 30 20:46:20.188919 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:20.188882    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"]
Oct 30 20:46:20.194741 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:20.194714    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-5291931c-kserve-854b59c7df-bxvz4"]
Oct 30 20:46:21.397410 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:21.397365    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1dc7f07-65ea-459e-b5ce-02fe26c9d193" path="/var/lib/kubelet/pods/f1dc7f07-65ea-459e-b5ce-02fe26c9d193/volumes"
Oct 30 20:46:30.808717 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:30.808683    5958 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-5295k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:46:30.809341 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:30.809291    5958 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-5295k" podUID="cc591cde-0da2-429d-8b99-a98f35abadd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:46:36.605047 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:46:36.605009    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:46:42.784245 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:42.784208    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49ac18a9-b45gg"]
Oct 30 20:46:42.784680 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:42.784665    5958 cpu_manager.go:414] "containerMap: removing container" podUID="ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b" containerName="revoke-keys"
Oct 30 20:46:42.784720 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:42.784681    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b" containerName="revoke-keys"
Oct 30 20:46:42.784720 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:42.784709    5958 cpu_manager.go:414] "containerMap: removing container" podUID="f1dc7f07-65ea-459e-b5ce-02fe26c9d193" containerName="main"
Oct 30 20:46:42.784720 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:42.784715    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1dc7f07-65ea-459e-b5ce-02fe26c9d193" containerName="main"
Oct 30 20:46:42.784813 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:42.784729    5958 cpu_manager.go:414] "containerMap: removing container" podUID="6228f70c-47e8-4e38-bff4-554be632356c" containerName="main"
Oct 30 20:46:42.784813 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:42.784734    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6228f70c-47e8-4e38-bff4-554be632356c" containerName="main"
Oct 30 20:46:42.784813 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:42.784809    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="6228f70c-47e8-4e38-bff4-554be632356c" containerName="main"
Oct 30 20:46:42.784906 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:42.784822    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b" containerName="revoke-keys"
Oct 30 20:46:42.784906 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:42.784832    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1dc7f07-65ea-459e-b5ce-02fe26c9d193" containerName="main"
Oct 30 20:46:42.789121 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:42.789100    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ac18a9-b45gg"
Oct 30 20:46:42.812534 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:42.812505    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49ac18a9-b45gg\" (UID: \"dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ac18a9-b45gg"
Oct 30 20:46:42.812674 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:42.812545    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49ac18a9-b45gg\" (UID: \"dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ac18a9-b45gg"
Oct 30 20:46:42.812674 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:42.812659    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmhtf\" (UniqueName: \"kubernetes.io/projected/dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4-kube-api-access-dmhtf\") pod \"maas-api-revoke-sub-49ac18a9-b45gg\" (UID: \"dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ac18a9-b45gg"
Oct 30 20:46:42.855618 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:42.855590    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:46:42.905406 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:42.905366    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:46:42.905593 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:42.905430    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:46:42.905593 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:42.905521    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpsc8\""
Oct 30 20:46:42.907037 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:42.907015    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49ac18a9-b45gg"]
Oct 30 20:46:42.914006 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:42.913980    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmhtf\" (UniqueName: \"kubernetes.io/projected/dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4-kube-api-access-dmhtf\") pod \"maas-api-revoke-sub-49ac18a9-b45gg\" (UID: \"dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ac18a9-b45gg"
Oct 30 20:46:42.914107 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:42.914063    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49ac18a9-b45gg\" (UID: \"dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ac18a9-b45gg"
Oct 30 20:46:42.914107 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:42.914089    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49ac18a9-b45gg\" (UID: \"dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ac18a9-b45gg"
Oct 30 20:46:42.914880 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:42.914862    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49ac18a9-b45gg\" (UID: \"dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ac18a9-b45gg"
Oct 30 20:46:42.916516 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:42.916500    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49ac18a9-b45gg\" (UID: \"dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ac18a9-b45gg"
Oct 30 20:46:42.975245 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:42.975217    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmhtf\" (UniqueName: \"kubernetes.io/projected/dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4-kube-api-access-dmhtf\") pod \"maas-api-revoke-sub-49ac18a9-b45gg\" (UID: \"dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ac18a9-b45gg"
Oct 30 20:46:43.101763 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:43.101740    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ac18a9-b45gg"
Oct 30 20:46:43.186915 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:43.186886    5958 scope.go:122] "RemoveContainer" containerID="e2205333cea1bf5eabc28a564d8dfc43d5f5dab740341030367dcd593e5eb61f"
Oct 30 20:46:43.472916 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:43.472895    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49ac18a9-b45gg"]
Oct 30 20:46:43.475151 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:46:43.475127    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb2d6e0_d995_4e7b_a4de_f886c7f49dc4.slice/crio-907680499d19db17362990b8e9d5483729957e036159394f5c4e668cef777b01 WatchSource:0}: Error finding container 907680499d19db17362990b8e9d5483729957e036159394f5c4e668cef777b01: Status 404 returned error can't find the container with id 907680499d19db17362990b8e9d5483729957e036159394f5c4e668cef777b01
Oct 30 20:46:44.161963 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:44.161930    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ac18a9-b45gg" event={"ID":"dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4","Type":"ContainerStarted","Data":"9cbd8b3fc9d063b757baf7201ec233e07bdf5754dcf749acac441cacd9a4faaf"}
Oct 30 20:46:44.162325 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:44.161982    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ac18a9-b45gg" event={"ID":"dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4","Type":"ContainerStarted","Data":"907680499d19db17362990b8e9d5483729957e036159394f5c4e668cef777b01"}
Oct 30 20:46:44.279419 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:44.279373    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ac18a9-b45gg" podStartSLOduration=2.279360408 podStartE2EDuration="2.279360408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:44.277597903 +0000 UTC m=+3843.009813291" watchObservedRunningTime="2026-10-30 20:46:44.279360408 +0000 UTC m=+3843.011575795"
Oct 30 20:46:45.168596 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:45.168560    5958 generic.go:356] "Generic (PLEG): container finished" podID="dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4" containerID="9cbd8b3fc9d063b757baf7201ec233e07bdf5754dcf749acac441cacd9a4faaf" exitCode=0
Oct 30 20:46:45.169068 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:45.168631    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ac18a9-b45gg" event={"ID":"dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4","Type":"ContainerDied","Data":"9cbd8b3fc9d063b757baf7201ec233e07bdf5754dcf749acac441cacd9a4faaf"}
Oct 30 20:46:46.305973 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:46.305950    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ac18a9-b45gg"
Oct 30 20:46:46.345539 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:46.345513    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4-maas-api-openshift-ca\") pod \"dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4\" (UID: \"dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4\") "
Oct 30 20:46:46.345653 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:46.345582    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4-kube-api-access-dmhtf\" (UniqueName: \"kubernetes.io/projected/dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4-kube-api-access-dmhtf\") pod \"dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4\" (UID: \"dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4\") "
Oct 30 20:46:46.345653 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:46.345636    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4-maas-api-xks-ca\") pod \"dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4\" (UID: \"dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4\") "
Oct 30 20:46:46.345845 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:46.345824    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4-maas-api-openshift-ca" pod "dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4" (UID: "dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:46:46.347802 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:46.347771    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4-maas-api-xks-ca" pod "dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4" (UID: "dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:46:46.347919 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:46.347895    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4-kube-api-access-dmhtf" pod "dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4" (UID: "dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4"). InnerVolumeSpecName "kube-api-access-dmhtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:46:46.447282 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:46.447182    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4-maas-api-openshift-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:46.447282 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:46.447216    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmhtf\" (UniqueName: \"kubernetes.io/projected/dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4-kube-api-access-dmhtf\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:46.447282 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:46.447235    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4-maas-api-xks-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:46:47.179677 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:47.179645    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ac18a9-b45gg"
Oct 30 20:46:47.179861 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:47.179679    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ac18a9-b45gg" event={"ID":"dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4","Type":"ContainerDied","Data":"907680499d19db17362990b8e9d5483729957e036159394f5c4e668cef777b01"}
Oct 30 20:46:47.179861 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:47.179727    5958 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="907680499d19db17362990b8e9d5483729957e036159394f5c4e668cef777b01"
Oct 30 20:46:56.253860 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.253826    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz"]
Oct 30 20:46:56.254279 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.254265    5958 cpu_manager.go:414] "containerMap: removing container" podUID="dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4" containerName="revoke-keys"
Oct 30 20:46:56.254339 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.254280    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4" containerName="revoke-keys"
Oct 30 20:46:56.254379 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.254356    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4" containerName="revoke-keys"
Oct 30 20:46:56.257775 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.257756    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz"
Oct 30 20:46:56.278811 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.278784    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-xf2t6"]
Oct 30 20:46:56.282308 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.282290    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-xf2t6"
Oct 30 20:46:56.327114 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.327083    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 20:46:56.327298 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.327228    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 20:46:56.338122 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.338088    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a586afc3-ab72-481d-9a91-09d84c5523df-metrics-certs\") pod \"maas-api-5c48877c75-j9hhz\" (UID: \"a586afc3-ab72-481d-9a91-09d84c5523df\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz"
Oct 30 20:46:56.338265 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.338162    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed8fbe1e-510b-4060-bf00-9fed0b94aa6d-metrics-certs\") pod \"maas-api-595cc648c-xf2t6\" (UID: \"ed8fbe1e-510b-4060-bf00-9fed0b94aa6d\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xf2t6"
Oct 30 20:46:56.338265 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.338247    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed8fbe1e-510b-4060-bf00-9fed0b94aa6d-openshift-service-ca\") pod \"maas-api-595cc648c-xf2t6\" (UID: \"ed8fbe1e-510b-4060-bf00-9fed0b94aa6d\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xf2t6"
Oct 30 20:46:56.338371 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.338292    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a586afc3-ab72-481d-9a91-09d84c5523df-openshift-service-ca\") pod \"maas-api-5c48877c75-j9hhz\" (UID: \"a586afc3-ab72-481d-9a91-09d84c5523df\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz"
Oct 30 20:46:56.338418 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.338374    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a586afc3-ab72-481d-9a91-09d84c5523df-maas-api-tls\") pod \"maas-api-5c48877c75-j9hhz\" (UID: \"a586afc3-ab72-481d-9a91-09d84c5523df\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz"
Oct 30 20:46:56.338474 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.338421    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed8fbe1e-510b-4060-bf00-9fed0b94aa6d-maas-api-tls\") pod \"maas-api-595cc648c-xf2t6\" (UID: \"ed8fbe1e-510b-4060-bf00-9fed0b94aa6d\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xf2t6"
Oct 30 20:46:56.338539 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.338473    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f7vl\" (UniqueName: \"kubernetes.io/projected/a586afc3-ab72-481d-9a91-09d84c5523df-kube-api-access-9f7vl\") pod \"maas-api-5c48877c75-j9hhz\" (UID: \"a586afc3-ab72-481d-9a91-09d84c5523df\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz"
Oct 30 20:46:56.338539 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.338524    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nq2p\" (UniqueName: \"kubernetes.io/projected/ed8fbe1e-510b-4060-bf00-9fed0b94aa6d-kube-api-access-6nq2p\") pod \"maas-api-595cc648c-xf2t6\" (UID: \"ed8fbe1e-510b-4060-bf00-9fed0b94aa6d\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xf2t6"
Oct 30 20:46:56.361625 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.361591    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 20:46:56.361768 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.361624    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 20:46:56.364577 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.364552    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz"]
Oct 30 20:46:56.367107 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.367083    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-xf2t6"]
Oct 30 20:46:56.412201 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.412184    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xtx52\""
Oct 30 20:46:56.439195 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.439173    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed8fbe1e-510b-4060-bf00-9fed0b94aa6d-openshift-service-ca\") pod \"maas-api-595cc648c-xf2t6\" (UID: \"ed8fbe1e-510b-4060-bf00-9fed0b94aa6d\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xf2t6"
Oct 30 20:46:56.439315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.439215    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a586afc3-ab72-481d-9a91-09d84c5523df-openshift-service-ca\") pod \"maas-api-5c48877c75-j9hhz\" (UID: \"a586afc3-ab72-481d-9a91-09d84c5523df\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz"
Oct 30 20:46:56.439315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.439257    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a586afc3-ab72-481d-9a91-09d84c5523df-maas-api-tls\") pod \"maas-api-5c48877c75-j9hhz\" (UID: \"a586afc3-ab72-481d-9a91-09d84c5523df\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz"
Oct 30 20:46:56.439315 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.439308    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed8fbe1e-510b-4060-bf00-9fed0b94aa6d-maas-api-tls\") pod \"maas-api-595cc648c-xf2t6\" (UID: \"ed8fbe1e-510b-4060-bf00-9fed0b94aa6d\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xf2t6"
Oct 30 20:46:56.439539 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.439435    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f7vl\" (UniqueName: \"kubernetes.io/projected/a586afc3-ab72-481d-9a91-09d84c5523df-kube-api-access-9f7vl\") pod \"maas-api-5c48877c75-j9hhz\" (UID: \"a586afc3-ab72-481d-9a91-09d84c5523df\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz"
Oct 30 20:46:56.439539 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.439500    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nq2p\" (UniqueName: \"kubernetes.io/projected/ed8fbe1e-510b-4060-bf00-9fed0b94aa6d-kube-api-access-6nq2p\") pod \"maas-api-595cc648c-xf2t6\" (UID: \"ed8fbe1e-510b-4060-bf00-9fed0b94aa6d\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xf2t6"
Oct 30 20:46:56.439655 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.439607    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a586afc3-ab72-481d-9a91-09d84c5523df-metrics-certs\") pod \"maas-api-5c48877c75-j9hhz\" (UID: \"a586afc3-ab72-481d-9a91-09d84c5523df\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz"
Oct 30 20:46:56.439714 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.439655    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed8fbe1e-510b-4060-bf00-9fed0b94aa6d-metrics-certs\") pod \"maas-api-595cc648c-xf2t6\" (UID: \"ed8fbe1e-510b-4060-bf00-9fed0b94aa6d\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xf2t6"
Oct 30 20:46:56.440123 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.440071    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a586afc3-ab72-481d-9a91-09d84c5523df-openshift-service-ca\") pod \"maas-api-5c48877c75-j9hhz\" (UID: \"a586afc3-ab72-481d-9a91-09d84c5523df\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz"
Oct 30 20:46:56.440259 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.440100    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed8fbe1e-510b-4060-bf00-9fed0b94aa6d-openshift-service-ca\") pod \"maas-api-595cc648c-xf2t6\" (UID: \"ed8fbe1e-510b-4060-bf00-9fed0b94aa6d\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xf2t6"
Oct 30 20:46:56.442096 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.442065    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a586afc3-ab72-481d-9a91-09d84c5523df-maas-api-tls\") pod \"maas-api-5c48877c75-j9hhz\" (UID: \"a586afc3-ab72-481d-9a91-09d84c5523df\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz"
Oct 30 20:46:56.442230 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.442208    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a586afc3-ab72-481d-9a91-09d84c5523df-metrics-certs\") pod \"maas-api-5c48877c75-j9hhz\" (UID: \"a586afc3-ab72-481d-9a91-09d84c5523df\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz"
Oct 30 20:46:56.442392 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.442376    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed8fbe1e-510b-4060-bf00-9fed0b94aa6d-maas-api-tls\") pod \"maas-api-595cc648c-xf2t6\" (UID: \"ed8fbe1e-510b-4060-bf00-9fed0b94aa6d\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xf2t6"
Oct 30 20:46:56.442516 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.442496    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed8fbe1e-510b-4060-bf00-9fed0b94aa6d-metrics-certs\") pod \"maas-api-595cc648c-xf2t6\" (UID: \"ed8fbe1e-510b-4060-bf00-9fed0b94aa6d\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xf2t6"
Oct 30 20:46:56.500102 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.500073    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nq2p\" (UniqueName: \"kubernetes.io/projected/ed8fbe1e-510b-4060-bf00-9fed0b94aa6d-kube-api-access-6nq2p\") pod \"maas-api-595cc648c-xf2t6\" (UID: \"ed8fbe1e-510b-4060-bf00-9fed0b94aa6d\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xf2t6"
Oct 30 20:46:56.500344 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.500325    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f7vl\" (UniqueName: \"kubernetes.io/projected/a586afc3-ab72-481d-9a91-09d84c5523df-kube-api-access-9f7vl\") pod \"maas-api-5c48877c75-j9hhz\" (UID: \"a586afc3-ab72-481d-9a91-09d84c5523df\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz"
Oct 30 20:46:56.571252 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.571230    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz"
Oct 30 20:46:56.594391 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.594367    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-xf2t6"
Oct 30 20:46:56.952242 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.952172    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz"]
Oct 30 20:46:56.968857 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:46:56.968832    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda586afc3_ab72_481d_9a91_09d84c5523df.slice/crio-8cd783cef4662d00e1b60226df43613c799557700f638b88e550de6c97ed2c98 WatchSource:0}: Error finding container 8cd783cef4662d00e1b60226df43613c799557700f638b88e550de6c97ed2c98: Status 404 returned error can't find the container with id 8cd783cef4662d00e1b60226df43613c799557700f638b88e550de6c97ed2c98
Oct 30 20:46:56.970189 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:56.970172    5958 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:46:57.001657 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:57.001638    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-xf2t6"]
Oct 30 20:46:57.003660 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:46:57.003641    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8fbe1e_510b_4060_bf00_9fed0b94aa6d.slice/crio-3988ffcf7be4940522327ac9b1298b1f16cb2590fdceb6a312fd3aab02884b15 WatchSource:0}: Error finding container 3988ffcf7be4940522327ac9b1298b1f16cb2590fdceb6a312fd3aab02884b15: Status 404 returned error can't find the container with id 3988ffcf7be4940522327ac9b1298b1f16cb2590fdceb6a312fd3aab02884b15
Oct 30 20:46:57.229506 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:57.229424    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz" event={"ID":"a586afc3-ab72-481d-9a91-09d84c5523df","Type":"ContainerStarted","Data":"8cd783cef4662d00e1b60226df43613c799557700f638b88e550de6c97ed2c98"}
Oct 30 20:46:57.230563 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:57.230544    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-xf2t6" event={"ID":"ed8fbe1e-510b-4060-bf00-9fed0b94aa6d","Type":"ContainerStarted","Data":"3988ffcf7be4940522327ac9b1298b1f16cb2590fdceb6a312fd3aab02884b15"}
Oct 30 20:46:58.237151 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:58.237053    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz" event={"ID":"a586afc3-ab72-481d-9a91-09d84c5523df","Type":"ContainerStarted","Data":"09fefa3e402c4e9cc550a6ee23a2f7daff4ba788169a30339c38a52fd6eea72c"}
Oct 30 20:46:58.237668 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:58.237249    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz"
Oct 30 20:46:58.238682 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:58.238660    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-xf2t6" event={"ID":"ed8fbe1e-510b-4060-bf00-9fed0b94aa6d","Type":"ContainerStarted","Data":"a7a2e2c09eeeda391262538706a69c8324e60f09549fbfd689103a1fae4797f3"}
Oct 30 20:46:58.238773 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:58.238702    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-xf2t6"
Oct 30 20:46:58.355052 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:58.354987    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz" podStartSLOduration=1.501052447 podStartE2EDuration="2.354969927s" totalImagesPullingTime="853.91748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:58.354172323 +0000 UTC m=+3857.086387736" watchObservedRunningTime="2026-10-30 20:46:58.354969927 +0000 UTC m=+3857.087185317"
Oct 30 20:46:58.473071 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:46:58.473012    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-xf2t6" podStartSLOduration=1.56063158 podStartE2EDuration="2.472998897s" totalImagesPullingTime="912.367317ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:58.471008916 +0000 UTC m=+3857.203224307" watchObservedRunningTime="2026-10-30 20:46:58.472998897 +0000 UTC m=+3857.205214287"
Oct 30 20:47:04.248781 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:04.248753    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-xf2t6"
Oct 30 20:47:04.249263 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:04.249147    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz"
Oct 30 20:47:04.418691 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:04.418655    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz"]
Oct 30 20:47:04.418859 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:04.418825    5958 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz" podUID="a586afc3-ab72-481d-9a91-09d84c5523df" containerName="maas-api" containerID="cri-o://09fefa3e402c4e9cc550a6ee23a2f7daff4ba788169a30339c38a52fd6eea72c" gracePeriod=30
Oct 30 20:47:04.662414 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:04.662391    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz"
Oct 30 20:47:04.815788 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:04.815761    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a586afc3-ab72-481d-9a91-09d84c5523df-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a586afc3-ab72-481d-9a91-09d84c5523df-openshift-service-ca\") pod \"a586afc3-ab72-481d-9a91-09d84c5523df\" (UID: \"a586afc3-ab72-481d-9a91-09d84c5523df\") "
Oct 30 20:47:04.815934 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:04.815799    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a586afc3-ab72-481d-9a91-09d84c5523df-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a586afc3-ab72-481d-9a91-09d84c5523df-metrics-certs\") pod \"a586afc3-ab72-481d-9a91-09d84c5523df\" (UID: \"a586afc3-ab72-481d-9a91-09d84c5523df\") "
Oct 30 20:47:04.815934 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:04.815835    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a586afc3-ab72-481d-9a91-09d84c5523df-kube-api-access-9f7vl\" (UniqueName: \"kubernetes.io/projected/a586afc3-ab72-481d-9a91-09d84c5523df-kube-api-access-9f7vl\") pod \"a586afc3-ab72-481d-9a91-09d84c5523df\" (UID: \"a586afc3-ab72-481d-9a91-09d84c5523df\") "
Oct 30 20:47:04.815934 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:04.815853    5958 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a586afc3-ab72-481d-9a91-09d84c5523df-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a586afc3-ab72-481d-9a91-09d84c5523df-maas-api-tls\") pod \"a586afc3-ab72-481d-9a91-09d84c5523df\" (UID: \"a586afc3-ab72-481d-9a91-09d84c5523df\") "
Oct 30 20:47:04.816193 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:04.816162    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a586afc3-ab72-481d-9a91-09d84c5523df-openshift-service-ca" pod "a586afc3-ab72-481d-9a91-09d84c5523df" (UID: "a586afc3-ab72-481d-9a91-09d84c5523df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:47:04.816284 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:04.816264    5958 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a586afc3-ab72-481d-9a91-09d84c5523df-openshift-service-ca\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:04.817987 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:04.817964    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a586afc3-ab72-481d-9a91-09d84c5523df-kube-api-access-9f7vl" pod "a586afc3-ab72-481d-9a91-09d84c5523df" (UID: "a586afc3-ab72-481d-9a91-09d84c5523df"). InnerVolumeSpecName "kube-api-access-9f7vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:47:04.818095 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:04.817981    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a586afc3-ab72-481d-9a91-09d84c5523df-metrics-certs" pod "a586afc3-ab72-481d-9a91-09d84c5523df" (UID: "a586afc3-ab72-481d-9a91-09d84c5523df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:47:04.818095 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:04.818000    5958 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a586afc3-ab72-481d-9a91-09d84c5523df-maas-api-tls" pod "a586afc3-ab72-481d-9a91-09d84c5523df" (UID: "a586afc3-ab72-481d-9a91-09d84c5523df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:47:04.917345 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:04.917319    5958 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a586afc3-ab72-481d-9a91-09d84c5523df-metrics-certs\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:04.917345 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:04.917342    5958 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f7vl\" (UniqueName: \"kubernetes.io/projected/a586afc3-ab72-481d-9a91-09d84c5523df-kube-api-access-9f7vl\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:04.917528 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:04.917353    5958 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a586afc3-ab72-481d-9a91-09d84c5523df-maas-api-tls\") on node \"ip-10-0-142-240.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:47:05.275783 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:05.275699    5958 generic.go:356] "Generic (PLEG): container finished" podID="a586afc3-ab72-481d-9a91-09d84c5523df" containerID="09fefa3e402c4e9cc550a6ee23a2f7daff4ba788169a30339c38a52fd6eea72c" exitCode=0
Oct 30 20:47:05.275783 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:05.275745    5958 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz"
Oct 30 20:47:05.276206 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:05.275781    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz" event={"ID":"a586afc3-ab72-481d-9a91-09d84c5523df","Type":"ContainerDied","Data":"09fefa3e402c4e9cc550a6ee23a2f7daff4ba788169a30339c38a52fd6eea72c"}
Oct 30 20:47:05.276206 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:05.275840    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz" event={"ID":"a586afc3-ab72-481d-9a91-09d84c5523df","Type":"ContainerDied","Data":"8cd783cef4662d00e1b60226df43613c799557700f638b88e550de6c97ed2c98"}
Oct 30 20:47:05.276206 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:05.275865    5958 scope.go:122] "RemoveContainer" containerID="09fefa3e402c4e9cc550a6ee23a2f7daff4ba788169a30339c38a52fd6eea72c"
Oct 30 20:47:05.285780 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:05.285762    5958 scope.go:122] "RemoveContainer" containerID="09fefa3e402c4e9cc550a6ee23a2f7daff4ba788169a30339c38a52fd6eea72c"
Oct 30 20:47:05.286068 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:47:05.286047    5958 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09fefa3e402c4e9cc550a6ee23a2f7daff4ba788169a30339c38a52fd6eea72c\": container with ID starting with 09fefa3e402c4e9cc550a6ee23a2f7daff4ba788169a30339c38a52fd6eea72c not found: ID does not exist" containerID="09fefa3e402c4e9cc550a6ee23a2f7daff4ba788169a30339c38a52fd6eea72c"
Oct 30 20:47:05.286121 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:05.286077    5958 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09fefa3e402c4e9cc550a6ee23a2f7daff4ba788169a30339c38a52fd6eea72c"} err="failed to get container status \"09fefa3e402c4e9cc550a6ee23a2f7daff4ba788169a30339c38a52fd6eea72c\": rpc error: code = NotFound desc = could not find container \"09fefa3e402c4e9cc550a6ee23a2f7daff4ba788169a30339c38a52fd6eea72c\": container with ID starting with 09fefa3e402c4e9cc550a6ee23a2f7daff4ba788169a30339c38a52fd6eea72c not found: ID does not exist"
Oct 30 20:47:05.442034 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:05.442006    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz"]
Oct 30 20:47:05.445546 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:05.445523    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-j9hhz"]
Oct 30 20:47:06.604221 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:47:06.604187    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:47:07.398296 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:07.398253    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a586afc3-ab72-481d-9a91-09d84c5523df" path="/var/lib/kubelet/pods/a586afc3-ab72-481d-9a91-09d84c5523df/volumes"
Oct 30 20:47:12.051690 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:12.051656    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nksc8"]
Oct 30 20:47:12.101674 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:12.101645    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nksc8"]
Oct 30 20:47:13.397614 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:13.397576    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5c705b4-1d88-4a02-b59f-8f7bb9af6d43" path="/var/lib/kubelet/pods/d5c705b4-1d88-4a02-b59f-8f7bb9af6d43/volumes"
Oct 30 20:47:33.050592 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:33.050557    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zmvlw"]
Oct 30 20:47:33.100651 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:33.100620    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-zmvlw"]
Oct 30 20:47:33.397438 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:33.397406    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c224149-348e-48aa-ad3f-be2310c6f162" path="/var/lib/kubelet/pods/1c224149-348e-48aa-ad3f-be2310c6f162/volumes"
Oct 30 20:47:36.605205 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:47:36.605164    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:47:42.019561 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:42.019530    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 20:47:42.035729 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:47:42.035708    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ft6pg_023e1dd6-ffb9-48c8-a407-be580b23bddb/ovn-acl-logging/0.log"
Oct 30 20:48:06.606022 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:48:06.605973    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:48:11.121923 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:11.121889    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-79bfbfd85f-tvtpm_611350ed-85e8-4bfd-8501-aedf22c38ab6/authorino/0.log"
Oct 30 20:48:21.972621 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:21.972587    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-dd496665f-x67wb_07bb7383-2ee5-459c-9464-06668df1a41f/manager/0.log"
Oct 30 20:48:24.388472 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:24.388410    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-gct5q_bc91bfd0-25ad-40c0-b87b-f6a792372cf8/manager/0.log"
Oct 30 20:48:25.067641 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:25.067611    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb_16c731fa-8a91-4cd9-9f36-a3078026bbe3/copy-manifests/0.log"
Oct 30 20:48:25.140777 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:25.140751    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb_16c731fa-8a91-4cd9-9f36-a3078026bbe3/manager/0.log"
Oct 30 20:48:26.135198 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:26.135171    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-2fw7s_2543c3b9-ba23-4040-adbb-048caa38ca10/proxy/0.log"
Oct 30 20:48:27.333889 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:27.333860    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-xf2t6_ed8fbe1e-510b-4060-bf00-9fed0b94aa6d/maas-api/0.log"
Oct 30 20:48:27.863609 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:27.863579    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-6qmsf_ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b/revoke-keys/0.log"
Oct 30 20:48:28.087974 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:28.087948    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-bgls7_48bfa585-2d57-469c-ba55-8ac07b7100c5/revoke-keys/0.log"
Oct 30 20:48:28.330746 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:28.330723    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-49ac18a9-b45gg_dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4/revoke-keys/0.log"
Oct 30 20:48:30.971033 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:30.971003    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-79bfbfd85f-tvtpm_611350ed-85e8-4bfd-8501-aedf22c38ab6/authorino/0.log"
Oct 30 20:48:31.456573 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:31.456544    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-j2cqk_0db352c1-b58a-4214-84ef-b507c534184a/manager/0.log"
Oct 30 20:48:34.475265 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:34.475231    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64985d6c8c-7n6jf_83edebaa-d6c0-401b-930c-a75f91343c70/kube-auth-proxy/0.log"
Oct 30 20:48:35.202187 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:35.202158    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-5295k_cc591cde-0da2-429d-8b99-a98f35abadd9/istio-proxy/0.log"
Oct 30 20:48:36.011015 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:36.010987    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-ds7rd_5f1d3194-b173-4350-95c7-3bba8f068267/istio-proxy/0.log"
Oct 30 20:48:36.605265 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:48:36.605230    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:48:37.403964 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:37.403937    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-wgtq6_496d86d7-0610-4b19-803d-44ef7ce1fe6b/istio-proxy/0.log"
Oct 30 20:48:37.686593 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:37.686527    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-27d65_7bb44edd-1fa8-495d-872f-84e0156aedf0/payload-pre-processing/0.log"
Oct 30 20:48:37.907783 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:37.907754    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mlhzl_86400592-9707-4383-bac2-3f7d8e8eb053/payload-processing/0.log"
Oct 30 20:48:39.070880 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:39.070846    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh_0df903f0-1e53-4243-b2d9-77f470486d7e/main/0.log"
Oct 30 20:48:40.115182 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:40.115155    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295_f55f875c-cde6-43b0-bd95-ead043376d8f/main/0.log"
Oct 30 20:48:40.622251 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:40.622226    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n_6e20c97b-66ce-485b-ba8d-90c2d9e1fa76/main/0.log"
Oct 30 20:48:41.685158 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:41.685131    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-79bfbfd85f-tvtpm_611350ed-85e8-4bfd-8501-aedf22c38ab6/authorino/0.log"
Oct 30 20:48:43.229746 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:43.229716    5958 scope.go:122] "RemoveContainer" containerID="87da1707a7c71d20e10db578c478522ed82daa56f7e2207637ffa0b53a5cccda"
Oct 30 20:48:43.244364 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:43.244340    5958 scope.go:122] "RemoveContainer" containerID="0be0d3a86bb61940e42db38f40d6764e2c3b4dd551c19887283e096b4266f190"
Oct 30 20:48:51.361339 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:51.361268    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-dd496665f-x67wb_07bb7383-2ee5-459c-9464-06668df1a41f/manager/0.log"
Oct 30 20:48:53.347889 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:53.347856    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-gct5q_bc91bfd0-25ad-40c0-b87b-f6a792372cf8/manager/0.log"
Oct 30 20:48:54.086630 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:54.086598    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb_16c731fa-8a91-4cd9-9f36-a3078026bbe3/copy-manifests/0.log"
Oct 30 20:48:54.170978 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:54.170951    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bcb8cc7d5-8cmsb_16c731fa-8a91-4cd9-9f36-a3078026bbe3/manager/0.log"
Oct 30 20:48:55.189286 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:55.189256    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-2fw7s_2543c3b9-ba23-4040-adbb-048caa38ca10/proxy/0.log"
Oct 30 20:48:56.254902 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:56.254870    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-xf2t6_ed8fbe1e-510b-4060-bf00-9fed0b94aa6d/maas-api/0.log"
Oct 30 20:48:56.777638 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:56.777610    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-6qmsf_ffa653c7-81f6-49f8-9f30-d2ffc90a2f2b/revoke-keys/0.log"
Oct 30 20:48:57.009411 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:57.009384    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-bgls7_48bfa585-2d57-469c-ba55-8ac07b7100c5/revoke-keys/0.log"
Oct 30 20:48:57.238249 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:57.238216    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-49ac18a9-b45gg_dfb2d6e0-d995-4e7b-a4de-f886c7f49dc4/revoke-keys/0.log"
Oct 30 20:48:59.105844 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:59.105815    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-79bfbfd85f-tvtpm_611350ed-85e8-4bfd-8501-aedf22c38ab6/authorino/0.log"
Oct 30 20:48:59.582965 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:48:59.582938    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-j2cqk_0db352c1-b58a-4214-84ef-b507c534184a/manager/0.log"
Oct 30 20:49:02.271660 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:49:02.271624    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64985d6c8c-7n6jf_83edebaa-d6c0-401b-930c-a75f91343c70/kube-auth-proxy/0.log"
Oct 30 20:49:03.022080 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:49:03.022049    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-5295k_cc591cde-0da2-429d-8b99-a98f35abadd9/istio-proxy/0.log"
Oct 30 20:49:03.864575 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:49:03.864548    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-ds7rd_5f1d3194-b173-4350-95c7-3bba8f068267/istio-proxy/0.log"
Oct 30 20:49:05.300933 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:49:05.300907    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-wgtq6_496d86d7-0610-4b19-803d-44ef7ce1fe6b/istio-proxy/0.log"
Oct 30 20:49:05.626757 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:49:05.626732    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-27d65_7bb44edd-1fa8-495d-872f-84e0156aedf0/payload-pre-processing/0.log"
Oct 30 20:49:05.856255 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:49:05.856233    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mlhzl_86400592-9707-4383-bac2-3f7d8e8eb053/payload-processing/0.log"
Oct 30 20:49:06.605270 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:49:06.605238    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:49:06.748704 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:49:06.748662    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-4cvsh_0df903f0-1e53-4243-b2d9-77f470486d7e/main/0.log"
Oct 30 20:49:07.779799 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:49:07.779765    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8q5295_f55f875c-cde6-43b0-bd95-ead043376d8f/main/0.log"
Oct 30 20:49:08.288277 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:49:08.288251    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-j8j5n_6e20c97b-66ce-485b-ba8d-90c2d9e1fa76/main/0.log"
Oct 30 20:49:36.605724 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:49:36.605685    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:49:38.940472 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:49:38.940420    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mlhzl_86400592-9707-4383-bac2-3f7d8e8eb053/payload-processing/0.log"
Oct 30 20:49:39.176819 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:49:39.176791    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mlhzl_86400592-9707-4383-bac2-3f7d8e8eb053/payload-processing/0.log"
Oct 30 20:50:06.604063 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:50:06.604029    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:50:18.226464 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:18.226428    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mlhzl_86400592-9707-4383-bac2-3f7d8e8eb053/payload-processing/0.log"
Oct 30 20:50:18.453260 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:18.453235    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mlhzl_86400592-9707-4383-bac2-3f7d8e8eb053/payload-processing/0.log"
Oct 30 20:50:27.861837 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:27.861807    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8t48c_93cfc69d-646b-4910-972d-04c9076c6c3c/global-pull-secret-syncer/0.log"
Oct 30 20:50:31.629091 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:31.629058    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rwbsr_ba9ed4cb-13c2-4ead-b549-5469e2724f6c/konnectivity-agent/0.log"
Oct 30 20:50:33.498288 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:33.498254    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-240.us-west-1.compute.internal_84c50bb941905481e1c090101140d6a9/haproxy/0.log"
Oct 30 20:50:36.612136 ip-10-0-142-240 kubenswrapper[5958]: E1030 20:50:36.612100    5958 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t4b2n" containerName="extractor"
Oct 30 20:50:37.107619 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.107589    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-79bfbfd85f-tvtpm_611350ed-85e8-4bfd-8501-aedf22c38ab6/authorino/0.log"
Oct 30 20:50:37.444942 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.444875    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-j2cqk_0db352c1-b58a-4214-84ef-b507c534184a/manager/0.log"
Oct 30 20:50:37.775144 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.775069    5958 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t2ndh/perf-node-gather-daemonset-d5hm9"]
Oct 30 20:50:37.775649 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.775630    5958 cpu_manager.go:414] "containerMap: removing container" podUID="a586afc3-ab72-481d-9a91-09d84c5523df" containerName="maas-api"
Oct 30 20:50:37.775649 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.775649    5958 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a586afc3-ab72-481d-9a91-09d84c5523df" containerName="maas-api"
Oct 30 20:50:37.775772 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.775732    5958 memory_manager.go:365] "RemoveStaleState removing state" podUID="a586afc3-ab72-481d-9a91-09d84c5523df" containerName="maas-api"
Oct 30 20:50:37.779181 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.779163    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-d5hm9"
Oct 30 20:50:37.808508 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.808482    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9be8107-251a-4589-a5d7-a6c311135d82-lib-modules\") pod \"perf-node-gather-daemonset-d5hm9\" (UID: \"f9be8107-251a-4589-a5d7-a6c311135d82\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-d5hm9"
Oct 30 20:50:37.808619 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.808517    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9be8107-251a-4589-a5d7-a6c311135d82-sys\") pod \"perf-node-gather-daemonset-d5hm9\" (UID: \"f9be8107-251a-4589-a5d7-a6c311135d82\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-d5hm9"
Oct 30 20:50:37.808671 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.808619    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92v5q\" (UniqueName: \"kubernetes.io/projected/f9be8107-251a-4589-a5d7-a6c311135d82-kube-api-access-92v5q\") pod \"perf-node-gather-daemonset-d5hm9\" (UID: \"f9be8107-251a-4589-a5d7-a6c311135d82\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-d5hm9"
Oct 30 20:50:37.808671 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.808661    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9be8107-251a-4589-a5d7-a6c311135d82-podres\") pod \"perf-node-gather-daemonset-d5hm9\" (UID: \"f9be8107-251a-4589-a5d7-a6c311135d82\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-d5hm9"
Oct 30 20:50:37.808746 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.808694    5958 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9be8107-251a-4589-a5d7-a6c311135d82-proc\") pod \"perf-node-gather-daemonset-d5hm9\" (UID: \"f9be8107-251a-4589-a5d7-a6c311135d82\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-d5hm9"
Oct 30 20:50:37.831406 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.831366    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t2ndh\"/\"openshift-service-ca.crt\""
Oct 30 20:50:37.831598 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.831581    5958 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t2ndh\"/\"kube-root-ca.crt\""
Oct 30 20:50:37.876258 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.876233    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t2ndh/perf-node-gather-daemonset-d5hm9"]
Oct 30 20:50:37.881434 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.881413    5958 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t2ndh\"/\"default-dockercfg-zhp6b\""
Oct 30 20:50:37.909550 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.909526    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9be8107-251a-4589-a5d7-a6c311135d82-podres\") pod \"perf-node-gather-daemonset-d5hm9\" (UID: \"f9be8107-251a-4589-a5d7-a6c311135d82\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-d5hm9"
Oct 30 20:50:37.909639 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.909560    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9be8107-251a-4589-a5d7-a6c311135d82-proc\") pod \"perf-node-gather-daemonset-d5hm9\" (UID: \"f9be8107-251a-4589-a5d7-a6c311135d82\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-d5hm9"
Oct 30 20:50:37.909686 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.909642    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9be8107-251a-4589-a5d7-a6c311135d82-proc\") pod \"perf-node-gather-daemonset-d5hm9\" (UID: \"f9be8107-251a-4589-a5d7-a6c311135d82\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-d5hm9"
Oct 30 20:50:37.909686 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.909649    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9be8107-251a-4589-a5d7-a6c311135d82-lib-modules\") pod \"perf-node-gather-daemonset-d5hm9\" (UID: \"f9be8107-251a-4589-a5d7-a6c311135d82\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-d5hm9"
Oct 30 20:50:37.909686 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.909658    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9be8107-251a-4589-a5d7-a6c311135d82-podres\") pod \"perf-node-gather-daemonset-d5hm9\" (UID: \"f9be8107-251a-4589-a5d7-a6c311135d82\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-d5hm9"
Oct 30 20:50:37.909784 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.909690    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9be8107-251a-4589-a5d7-a6c311135d82-sys\") pod \"perf-node-gather-daemonset-d5hm9\" (UID: \"f9be8107-251a-4589-a5d7-a6c311135d82\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-d5hm9"
Oct 30 20:50:37.909784 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.909735    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9be8107-251a-4589-a5d7-a6c311135d82-lib-modules\") pod \"perf-node-gather-daemonset-d5hm9\" (UID: \"f9be8107-251a-4589-a5d7-a6c311135d82\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-d5hm9"
Oct 30 20:50:37.909784 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.909740    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9be8107-251a-4589-a5d7-a6c311135d82-sys\") pod \"perf-node-gather-daemonset-d5hm9\" (UID: \"f9be8107-251a-4589-a5d7-a6c311135d82\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-d5hm9"
Oct 30 20:50:37.909784 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.909754    5958 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92v5q\" (UniqueName: \"kubernetes.io/projected/f9be8107-251a-4589-a5d7-a6c311135d82-kube-api-access-92v5q\") pod \"perf-node-gather-daemonset-d5hm9\" (UID: \"f9be8107-251a-4589-a5d7-a6c311135d82\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-d5hm9"
Oct 30 20:50:37.970459 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:37.970422    5958 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92v5q\" (UniqueName: \"kubernetes.io/projected/f9be8107-251a-4589-a5d7-a6c311135d82-kube-api-access-92v5q\") pod \"perf-node-gather-daemonset-d5hm9\" (UID: \"f9be8107-251a-4589-a5d7-a6c311135d82\") " pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-d5hm9"
Oct 30 20:50:38.091691 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:38.091666    5958 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-d5hm9"
Oct 30 20:50:38.462080 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:38.462019    5958 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t2ndh/perf-node-gather-daemonset-d5hm9"]
Oct 30 20:50:38.464347 ip-10-0-142-240 kubenswrapper[5958]: W1030 20:50:38.464323    5958 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf9be8107_251a_4589_a5d7_a6c311135d82.slice/crio-fbdd5bc18e3f0e19da2c1816ddd94af7955321621940b788aafd50e6f1390611 WatchSource:0}: Error finding container fbdd5bc18e3f0e19da2c1816ddd94af7955321621940b788aafd50e6f1390611: Status 404 returned error can't find the container with id fbdd5bc18e3f0e19da2c1816ddd94af7955321621940b788aafd50e6f1390611
Oct 30 20:50:39.350688 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:39.350652    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-d5hm9" event={"ID":"f9be8107-251a-4589-a5d7-a6c311135d82","Type":"ContainerStarted","Data":"4b2b4af6c85bafe6e052ff7da40a64f84fe1c91cf6680e45236e7910f6d23333"}
Oct 30 20:50:39.351071 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:39.350708    5958 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-d5hm9" event={"ID":"f9be8107-251a-4589-a5d7-a6c311135d82","Type":"ContainerStarted","Data":"fbdd5bc18e3f0e19da2c1816ddd94af7955321621940b788aafd50e6f1390611"}
Oct 30 20:50:39.351071 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:39.350758    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-d5hm9"
Oct 30 20:50:39.467217 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:39.467165    5958 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-d5hm9" podStartSLOduration=2.467153282 podStartE2EDuration="2.467153282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:39.465587873 +0000 UTC m=+4078.197803263" watchObservedRunningTime="2026-10-30 20:50:39.467153282 +0000 UTC m=+4078.199368670"
Oct 30 20:50:42.488880 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:42.488846    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f252503f-59b8-47c1-9ff8-f4f704a159d4/alertmanager/0.log"
Oct 30 20:50:42.909728 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:42.909700    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f252503f-59b8-47c1-9ff8-f4f704a159d4/config-reloader/0.log"
Oct 30 20:50:43.275999 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:43.275914    5958 scope.go:122] "RemoveContainer" containerID="2358dc4266884f9504e747bf800cbf8a64fc03043074b714043357eddaf611d0"
Oct 30 20:50:43.329738 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:43.329719    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f252503f-59b8-47c1-9ff8-f4f704a159d4/kube-rbac-proxy-web/0.log"
Oct 30 20:50:43.749705 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:43.749680    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f252503f-59b8-47c1-9ff8-f4f704a159d4/kube-rbac-proxy/0.log"
Oct 30 20:50:44.170710 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:44.170687    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f252503f-59b8-47c1-9ff8-f4f704a159d4/kube-rbac-proxy-metric/0.log"
Oct 30 20:50:44.591863 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:44.591836    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f252503f-59b8-47c1-9ff8-f4f704a159d4/prom-label-proxy/0.log"
Oct 30 20:50:45.013551 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:45.013488    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f252503f-59b8-47c1-9ff8-f4f704a159d4/init-config-reloader/0.log"
Oct 30 20:50:45.049938 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:45.049906    5958 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bgls7"]
Oct 30 20:50:45.099800 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:45.099772    5958 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bgls7"]
Oct 30 20:50:45.367311 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:45.367291    5958 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t2ndh/perf-node-gather-daemonset-d5hm9"
Oct 30 20:50:45.397512 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:45.397052    5958 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48bfa585-2d57-469c-ba55-8ac07b7100c5" path="/var/lib/kubelet/pods/48bfa585-2d57-469c-ba55-8ac07b7100c5/volumes"
Oct 30 20:50:45.971750 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:45.971721    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-56t97_419b608e-bfa8-4d2b-bacd-08c286fb7707/kube-state-metrics/0.log"
Oct 30 20:50:46.386625 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:46.386594    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-56t97_419b608e-bfa8-4d2b-bacd-08c286fb7707/kube-rbac-proxy-main/0.log"
Oct 30 20:50:46.810807 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:46.810736    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-56t97_419b608e-bfa8-4d2b-bacd-08c286fb7707/kube-rbac-proxy-self/0.log"
Oct 30 20:50:47.240469 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:47.240421    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-8698dc55b4-pwqbt_154facaf-4163-41fb-a6d9-fb04a5d8e12e/metrics-server/0.log"
Oct 30 20:50:47.664123 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:47.664084    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-p527g_6f405174-11f7-414c-a940-2a63c8989f3a/monitoring-plugin/0.log"
Oct 30 20:50:50.649931 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:50.649907    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t586q_56bbeb6c-dce1-42dc-9291-615c39153a7d/node-exporter/0.log"
Oct 30 20:50:51.068419 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:51.068396    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t586q_56bbeb6c-dce1-42dc-9291-615c39153a7d/kube-rbac-proxy/0.log"
Oct 30 20:50:51.487594 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:51.487524    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t586q_56bbeb6c-dce1-42dc-9291-615c39153a7d/init-textfile/0.log"
Oct 30 20:50:51.912793 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:51.912766    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wd2gl_a94cd52c-5a75-45c0-9669-587b56e8fc70/kube-rbac-proxy-main/0.log"
Oct 30 20:50:52.332508 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:52.332480    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wd2gl_a94cd52c-5a75-45c0-9669-587b56e8fc70/kube-rbac-proxy-self/0.log"
Oct 30 20:50:52.755383 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:52.755307    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wd2gl_a94cd52c-5a75-45c0-9669-587b56e8fc70/openshift-state-metrics/0.log"
Oct 30 20:50:53.194618 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:53.194588    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_041ef390-b8bb-43e8-9adf-6c074e1a13b0/prometheus/0.log"
Oct 30 20:50:53.614294 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:53.614265    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_041ef390-b8bb-43e8-9adf-6c074e1a13b0/config-reloader/0.log"
Oct 30 20:50:54.043373 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:54.043284    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_041ef390-b8bb-43e8-9adf-6c074e1a13b0/thanos-sidecar/0.log"
Oct 30 20:50:54.508068 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:54.508033    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_041ef390-b8bb-43e8-9adf-6c074e1a13b0/kube-rbac-proxy-web/0.log"
Oct 30 20:50:54.926769 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:54.926744    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_041ef390-b8bb-43e8-9adf-6c074e1a13b0/kube-rbac-proxy/0.log"
Oct 30 20:50:55.346880 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:55.346854    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_041ef390-b8bb-43e8-9adf-6c074e1a13b0/kube-rbac-proxy-thanos/0.log"
Oct 30 20:50:55.767908 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:55.767847    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_041ef390-b8bb-43e8-9adf-6c074e1a13b0/init-config-reloader/0.log"
Oct 30 20:50:57.461361 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:57.461335    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-596848ccc9-tsm2b_91ef5bd9-9316-418a-a989-8557a38e0058/telemeter-client/0.log"
Oct 30 20:50:57.881666 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:57.881633    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-596848ccc9-tsm2b_91ef5bd9-9316-418a-a989-8557a38e0058/reload/0.log"
Oct 30 20:50:58.301340 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:58.301267    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-596848ccc9-tsm2b_91ef5bd9-9316-418a-a989-8557a38e0058/kube-rbac-proxy/0.log"
Oct 30 20:50:58.726798 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:58.726772    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f5bcd9b49-9bcsl_25a55956-200b-4ca6-ad5e-050caf7116e4/thanos-query/0.log"
Oct 30 20:50:59.147099 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:59.147063    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f5bcd9b49-9bcsl_25a55956-200b-4ca6-ad5e-050caf7116e4/kube-rbac-proxy-web/0.log"
Oct 30 20:50:59.567523 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:59.567496    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f5bcd9b49-9bcsl_25a55956-200b-4ca6-ad5e-050caf7116e4/kube-rbac-proxy/0.log"
Oct 30 20:50:59.987202 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:50:59.987135    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f5bcd9b49-9bcsl_25a55956-200b-4ca6-ad5e-050caf7116e4/prom-label-proxy/0.log"
Oct 30 20:51:00.411845 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:51:00.411773    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f5bcd9b49-9bcsl_25a55956-200b-4ca6-ad5e-050caf7116e4/kube-rbac-proxy-rules/0.log"
Oct 30 20:51:00.830805 ip-10-0-142-240 kubenswrapper[5958]: I1030 20:51:00.830776    5958 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f5bcd9b49-9bcsl_25a55956-200b-4ca6-ad5e-050caf7116e4/kube-rbac-proxy-metrics/0.log"